Troubleshooting CRM, Kerberos, and the Error 401.1 – Unauthorized: Access is denied

You install Microsoft Dynamics CRM 4.0 and follow the instructions to the best of your ability only to find that when you try to access the CRM web site, it prompts you for a password three times then you get the message:

You are not authorized to view this page.
HTTP Error 401.1 – Unauthorized: Access is denied due to invalid credentials.

This has got to be the #1 most common error people run into when installing Microsoft Dynamics CRM 4.0. In all of the production, testing, and development environments I have set up, the 401.1 error is the one I have spent the most time troubleshooting. It is ALWAYS caused by a problem with Kerberos and double-hop authentication. The trouble is that there are SO MANY ways you can break Kerberos. Over the weekend, I went through another marathon troubleshooting session to resolve yet another Kerberos issue. This time it was in our Sandbox (testing) hosting environment. It came down to an incorrect SPN this time.

Service Principal Names (SPNs) and Delegation
For Kerberos double-hop authentication to work, you must have the proper SPNs configured and you must configure delegation for the computer account or service account that the service runs under. For CRM, this is the identity that your CRM web site’s application pool uses.

The SPNs you need to configure for Kerberos double-hop authentication to work properly are set up as follows:

setspn –A HTTP/servername:5555 domain\serviceusername_or_computername
setspn –A HTTP/servername.company.com domain\serviceusername_or_computername

If you configure the application pool to use an AD user account (service account) as its identity, you should use the name of that account in the above configuration. If you use Network Service, you should use the computer name (and you won’t need to prefix it with domain\). These commands add an SPN in Active Directory. You can also use ADSIEdit to configure this. Note that you need to use the port number for the short (NetBIOS) name version of the SPN but NOT for the FQDN version. This one got me recently.

Next, you need to allow delegation for the computer or service account using ADUC.

Incorrect or Missing SPN
You need to make sure that you have the correct SPN configured. Use setspn -L “computer_or_account_name” to list all SPNs for the service account and computer accounts.

Ex:
setspn -L CRMSERVERNAME
setspn -L Domain\service_account_name

In a recent troubleshooting experience, I had the port number added to the FQDN SPN and I got the following misleading error in the event log:

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 4
Date: 1/20/2009
Time: 1:55:03 PM
User: N/A
Computer: ONCCRM01
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the server host/crmdiscoveryserver.domain.net. The target name used was HTTP/crmdiscoveryserver.domain.net. This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (MSCRMHOST.NET), and the client realm. Please contact your system administrator.
For more information, see Help and Support Center at

All of the info I found while searching indicated that the problem was due to the next issue, Duplicate SPNs, when in reality, it was because Kerberos was looking for an SPN without the TCP port and I had configured it with the TCP port.

Duplicate SPNs
Duplicate SPNs can be difficult to resolve if you don’t know how. 

Knowledgebase Article Windows 2000 Server Prompts Domain User for Credentials describes how to use LDP and Adsiedit.msc to troubleshoot and fix a duplicate SPN problem.  When you run LDP, make your search string http/* to return all SPNs configured for web sites.  You can narrow that down further to http/computername* if the former string yields too many results.  Do the same for host/* and MSSQLServer/* just in case.

Event ID 11 — Service Principal Name Configuration
Note the important comment that the “setspn -X” option is only available in Windows Server 2008, not 2003.  If you have Server 2008, this is much simpler than using LDP which can be daunting to those unfamiliar with AD and LDAP.

“Negotiate,NTLM” not configured on the web site and/or virtual directories
Your web site must have Negotiate/NTLM configured or the client and server will not negotiate Kerberos. This is the default setting but any number of things can change this configuration. The following article discusses how to configure this parameter using adsutil. I prefer to use the Metabase Explorer in the IIS Resource kit because it lets me see all of the configuration parameters and edit them directly. Either way will work.

Error message when you try to access the Microsoft Dynamics CRM Web site: “You are not authorized to view this page”

Keepalives not enabled on the CRM web site
Keepalives are turned on by default in IIS. If they are turned off, Kerberos will break. This is the most difficult issue I have ever had to troubleshoot related to the 401.1 error in CRM. I don’t know how it got turned off but the only thing that led me to the fix was to analyze the traffic with Netmon then use IIS Diagnostics to troubleshoot Kerberos. Use Metabase Explorer or adsutil to examine and change the keepalive setting.

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to FurlAdd to Newsvine

Cloud Computing Pervasive In 2009?

Let’s say you and I are starting a business and we’re making plans to launch a great new piece of software that will change the world. Should we be thinking cloud platform? Will our software product/service be obsolete by the time we finish it if we don’t build it on Google or Microsoft or Amazon’s cloud? The answer to these questions depend a lot on what software we’re trying to build. But I’d argue that it depends even more on our customers and what they want.

If you are a technocrat, you probably do what I do all the time and underestimate how long it will take for the the business crowd to adopt new technology. There are lots of reasons why businesses still haven’t adopted SaaS and don’t even care to learn about what “cloud computing” means. One reason that SaaS is still not pervasive is the lack of single-sign-on or at least some common or easily federated directory service. If I get 10 different pieces of business software from 10 different vendors then all of my employees have 10 different user logins and 10 different passwords to maintain and remember.  This won’t fly in the enterprise. Another big problem is the question of risk and trust. “Do I trust this hosted software company to maintain and secure my data?” Do I trust 10 different vendors to each maintain my data.

I gave a presentation during Microsoft Dynamics CRM Incubation week back in mid-December and I showed this graphic:

 
Technology Evolution vs. Customer Adoption

Technology Evolution vs. Customer Adoption

This is not exact science.  This is really my opinion of the approximate periods where each category of computing platform reached a critical mass in terms of CUSTOMER adoption starting with the client-server days (I’m old but I was still in college when mainframes/terminals were all the rage).  I should really back this up with sound research but my goal is to illustrate a point, not publish an article in a scientific journal. 

The point is that 2009 might be the year that Cloud Platform TECHNOLOGY is a real option but that certainly does NOT mean that cloud computing will be adopted by the majority of businesses any time soon.  So, if we want to start selling our software at the end of 2009 or early 2010, should we write it for Google’s AppEngine or build it completely on Microsoft’s Azure platform?  That’s probably not a good business move considering that SMBs haven’t moved to SaaS as a predominant platform yet, much less enterprises.  We would be a lot better off building our software so that we can offer it EITHER an on-premise OR a hosted (SaaS) model.  This is why my company developed the latest version of CourseMax on the Microsoft Dynamics CRM platform.  This is the Software + Services advantage, the “Power of Choice.”

Does this mean that we shouldn’t be considering cloud computing at all?  No way.  We should always be forward thinking.  We need to be ready to offer our customers what they want when they want it which means we better start working on it before they want it and we better start thinking about it before we start working on it.  Are there pieces of Google, Microsoft, or Amazon’s cloud platform we can start incorporating into our product now?  Sure, there may be some logical pieces of our architecture that we want to implement in the cloud.  Maybe we’ll use Microsoft Azure’s Internet Service Broker to handle data synchronization.

History has shown us a pretty natural cycle of platform adoption.  The future part of my graphic may be way off.  I’m not Nostradomus you know.  Perhaps the cloud is really an enhancement of SaaS that will actually accelerate SaaS adoption.  I think that may be partially true.  But there are still many things that need to shake out in cloud computing, standards to be built, interoperability, and much more before even the technology is ready for prime time.

One thing I am sure of is that businesses will be buying on-premise software for many years to come.  We should always learn from the past and look to the future.

Tell me what you think.  What’s your opinion?  Are you building software on a cloud platform already?

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to FurlAdd to Newsvine

Update Rollup 2 for Microsoft Dynamics CRM 4.0 Released

Microsoft has released Update Rollup 2 for CRM 4.0.  The rollup includes all of the Update Rollup 1 fixes and a ton of important new fixes.  These fixes address several important issues that I know many people were facing.  By my count, there are 111 new hotfixes in the rollup and 17 previously undocumented issues.

I was surprised (pleasantly) to see another rollup released so soon.  Update Rollup 1 came out back on November 24 (less than 2 months ago).  I guess that means that the CRM team has been hard at work cranking out fixes.  The fact that this happened over the holidays is even more impressive.

http://support.microsoft.com/kb/959419

There are a lot of fixes for the Outlook client and for workflows.  Here is my Top 20 list of the most important fixes (or at least the ones that cause the most problems for me and my colleagues):

Outlook Client Issues

  • 950372  You are prompted to enter your user name and password multiple times when you start Outlook, or when you click a Microsoft Dynamics CRM folder in Outlook, on a Windows Vista-based computer
  • 951432  You cannot go online again in the Microsoft Dynamics CRM 4.0 client for Outlook with Offline Access if you switch to a different network connection while you are offline
  • 952148  Error message the first time that you try to go online after you upgrade the Microsoft Dynamics CRM 3.0 client for Outlook to the Microsoft Dynamics CRM 4.0 client for Outlook with Offline Access: “An unidentified error occurred”
  • 956859  The Microsoft Dynamics CRM toolbar is disabled in Outlook after you lose the network connection for less than one minute
  • 957421  Outlook may crash when you synchronize Outlook with the Microsoft Dynamics CRM 4.0 client for Outlookli>

  • 958031  Error message when you go online in the Microsoft Dynamics CRM 4.0 client for Outlook over a low-bandwidth network or a high-latency network: “An unknown error has occurred”

Importing Data

  • 948987  Error message when you import data into Microsoft Dynamics CRM 4.0 by using the Data Import Wizard or start a new data migration by using Data Migration Manager: “Mismatched data delimiter: only one delimiter was found”
  • 949652  Error message when you try to import a CSV file into Microsoft Dynamics CRM 4.0: “The source data is not in the required format”
  • 953033  Error message when you try to import data that involves a picklist attribute in Microsoft Dynamics CRM 4.0: “Transform parse data”

Workflows and Async Service

  • 952322  A workflow rule does not work as expected when you perform the workflow rule in Microsoft Dynamics CRM 4.0
  • 953586  Error message when you try to start the Async service in Microsoft Dynamics CRM 4.0: “Exception while starting async service: MSCRMAsyncService”
  • 956237  The workflow remains in a waiting state after you create a workflow rule to update the status of an opportunity in Microsoft Dynamics CRM 4.0
  • 956282  Error message when you use the Microsoft Dynamics CRM 4.0 SDK to develop a Custom Workflow Activity assembly: “The Key cannot be NULL”
  • 956751  A workflow is triggered two times even though you set the corresponding workflow rule to run only when the activity status is Completed in Microsoft Dynamics CRM 4.0

Customizations

  • 946745  You cannot import the customization for an entity to a new system in Microsoft Dynamics CRM
  • 951909  Error message when you try to import a customization to another Microsoft Dynamics CRM 4.0 server: “Invalid name prefix”
  • 954322  Error message when you import some customizations to an upgraded installation of Microsoft Dynamics CRM 4.0: “Either the file could not be uploaded or Import failed”
  • Microsoft Dynamics CRM 4.0 cannot load custom pages that reference assemblies. This issue occurs if the assemblies are deployed to the following folder:
    <crmwebroot>\ISV\<app>\bin

Miscellaneous Errors

  • 958836  Error message when you run a custom report in Microsoft Dynamics CRM 4.0: “An error has occurred”
  • When you use impersonation in an Internet-Facing Deployment (IFD) environment, you receive the following error message: 0x80040204 Invalid user auth.

5-Days of Inspiration at Microsoft CRM Incubation Week

A couple weeks ago, I spent 5 busy Days in Reston, Virginia at the Microsoft Technology Center for Microsoft Dynamics CRM Incubation Week. This was a fantastic idea, hatched at Microsoft, to invite 5 startup companies to the MTC to spend 5 intensive days working on building a prototype and then, on the last day, to present their idea and prototype to a panel of VCs and the most influential writer in the CRM world, Paul Greenberg. This is one of the coolest things I’ve ever been involved in.  For more information about the event, check out these blogs from some of the people who were involved:

Day0: Microsoft Dynamics CRM Incubation Week
Sanjay Jain of Microsoft coordinated the event and made it all happen.  He blogged each day of the event starting with Day 0 above (I guess Sanjay thinks in Base-0)

5 ideas, 5 days and 5 businesses
Dave Drach, Managing Director, Emerging Business Team, Microsoft Corporation

Microsoft Gives Me A Happy Birthday: Entrepreneurs Rock!
Paul Greenberg, Writer and Influential CRM Thought Leader

CRM Incubation Week – Roundup
Girish Raja, Technical Evangelist for Microsoft Dynamics CRM at Microsoft

CRM Incubation Week Wrap Up
Jim Steger, Co-Author for two of the most popular (and best) books out there on Microsoft Dynamics CRM.  Jim was one of the advisors and was assigned to one team but everyone (including me) picked his brain.  How cool is it when the guy who wrote the book is right around the corner to answer the tough questions.  Jim does a nice job of summing up the event and referencing all of the participants in this post.

Incubation Week is a Hot Place to Be
Giuseppe (Joe) Zuccaro, The Marketing Consigliere

Increase Your Odds of Being Remarkable…
Jeremy Epstein, Former Microsoftie and now social media and online marketing guru for hire who presented on the topic of marketing and social media 

You Did All This in 5 Days…Really…No Way

My role was to be an advisor (one of 6) to the CRM teams. I also got to speak to the teams about my experience in building a business as a CRM ISV. On the last day, we went around the room and everyone gave their thoughts about the event. The responses were all extremely positive. The common thread from the VCs was that they were blown away with how much functionality and value these teams were able to develop and demonstrate with essentially 3 1/2 days to work on their prototypes. This, of course, didn’t surprise me much at all. CRM 4.0 is a great platform for building a business application. Of course there is a whole lot more work to do (more than anyone can even fathom who hasn’t built a commercial software application and business before) to turn that prototype into a real product and build a business around it. But building software on the CRM platform gives you a huge advantage in terms of how quickly you can bring a product to market.

What I found most interesting was that people outside the Microsoft Dynamics partner circle (the VCs who attended, Paul, and most of the business community) weren’t aware of how great the platform really is before they attended the event. The fact that Paul was so impressed really surprised me because he evaluates CRM platforms all the time and probably has a broader, more complete knowledge of their capabilities than anyone in the industry.  In his blog, he laid down this tongue-in-cheek challenge to Microsoft which I thought was hilarious:

“Now, for Microsoft to really do something with this, they have to showcase it beyond me writing about it. Be bold, Microsoft! Get out there! Develop a PR and marketing effort around this that’s exciting but not self-aggrandizing. You have five days.”

I don’ t think Microsoft has been quiet on CRM as a platform by any means.  It’s just that so many messages come out of Redmond that the message gets lost like a shout after a touchdown at the football game.  In contrast, all you hear about is salesforce.com’s force.com Platform as a Service (PaaS) play.  That’s really the only message coming from them so you hear it loud and clear.  So, even though Microsoft has a far better platform for a whole variety of reasons, not to mention the unbelievable support they give to their partner channel, you probably think of salesforce when you think of a CRM-based PaaS.  However, I would not be surprised if, by the end of 2009, the tables have turned completely. 

Microsoft is betting big on CRM.  CRM Services are even going to be part of the Azure cloud platform.  To me, Azure is a glimpse at the future of computing.  This cloud is a whole generation ahead of Amazon’s EC2 or anything else I’ve seen.

Giving and Getting Back Even More In Return

A lot of my friends, family, and colleagues couldn’t believe I took five days away from the business to do this but I can say wholeheartedly that it was well worth it.  There is much truth in the old cliche that when you give, you get much more back in return.  I got to know a bunch of phenomenal people and spend time with them in a really unique environment.  You really get to know someone when you are all working together in highly-effective teams to do something really extraordinary.   Microsoft gave me a Zune for being a presenter which I have to admit I haven’t had time to really use much yet.  It was extremely interesting and revealing to talk to the VCs and hear their feedback on the entrepreneurs’ presentations.  There is plenty of good advice out there about how to present to a VC but you don’t know how true it is until you witness real world examples of what flies with them and what doesn’t. 

But probably the biggest reward was the inspiration I got from these entrepreneurs who have big dreams and are chasing them with their heart, soul, blood, sweat, and tears.  I’ve been doing the entrepreneur thing for almost 10 years now and it’s easy to lose some of that energy after 10 years of 80-hour work weeks.  Hanging around these folks really refilled my tank.  To all of the enrepreneurs who participated, I hope you all achieve your dreams.  Don’t lose that spirit but temper it enough that you don’t forget your priorities like your family, your friends, and your health.  I hope your spouses are  as understanding and supportive as my wife is who just brought dinner into my office!

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to FurlAdd to Newsvine

A Primer on Multitenancy for Microsoft Dynamics CRM

There seems to be some confusion around how to configure and use multitenancy in Microsoft Dynamics CRM 4.0.  In simple terms, multitenancy means that more than one “tenant” shares a single implementation of a web application.  A tenant might be a single customer to a vendor that hosts the software for many customers.  Or, it might be a division or department in a large organization.  The big benefit of multitenancy is economy of scale.  You can install all of the software on a set of servers and serve several customers with one implementation.

For Microsoft CRM, multitenancy first became available with the release of Microsoft CRM 4.0 back in December 2007.  Prior to 4.0, you needed to install a separate copy of the software for every tenant. I’ve learned a great deal about the specifics of Microsoft CRM’s multitenant functionality over the last year from architecting CourseMax’s datacenter, consuming all of the documentation out there, bending the ear of many of my friends at Microsoft and some key partners, and doing consulting work for other ISVs who are in the process of launching apps on the CRM platform.  

Most of what you need to know to implement multitenancy can be found in documentation provided by Microsoft.  So why, you might ask, am I blogging about it.  Well, my experience has been that finding the right documentation is not easy because:

  • The available documentation contradicts itself (mostly cleared up in newer versions)
  • You have to know where to find the right information which is in different places
  • There is a large volume of very technical information to consume
  • It is difficult to pinpoint the information you need
  • There is not a good summary to give you the big picture.

I’ll follow up with additional posts and drill deeper into the various topics.  Hopefully I can demystify this topic, clear up some of the misconceptions, and help others get up and running on multi-tenant CRM more easily.

Installing CRM to Meet Multi-Tenancy Requirements

General Requirements:

  • Meet all of the normal requirements for installing CRM (not included here for the sake of brevity)
  • Enterprise version of CRM is required to support multitenancy
  • Install CRM Server Roles on separate servers as required to meet performance requirements (brevity note applies here as well)
  • Deploy (provision) CRM organizations using CRM Deployment Manager or programmatically using the deployment service

Requirements for Off-Premise access to CRM

  • Configure the Internet Facing Deployment (“IFD”) option either at time of install or after install using the IFD Config tool
  • Obtain and install a wildcard SSL certificate on the CRM Application Server(s)
  • Configure hostnames in DNS or set up wildcard DNS for off-premise FQDNs

Completing off-premise access requirements will enable forms-based authentication.  Forms-based authentication presents a login form to the end user as shown in the image below.  When the user logs in through this form, the CRM server then proxies the authentication to ActiveDirectory for the user.

CRM IFD Login Page

How to Deploy (Provision) Organizations

You can deploy organizations manually using the CRM Deployment Manager MMC snap-in.  Deployment Manager is installed on servers where you install the CRM Application Server role.  You must provide the following information to create a new organization:

  • Display name (Shown on the header of the CRM application)
  • Name (this is the unique organization name)
  • ISO currency code
  • SQL Collation
  • SQL Server (Selected on a screen that follows)
  • Report Server URL (Entered on a screen that follows)

Deployment Manager - New Organization

After you have input the above information, the wizard will validate the input and attempt to connect to the SQL Server and SRS.  If validation is successful, you can select finish to create the organization.  The wizard will:

  • Create a New Database named “Name_MSCRM” on the specified SQL Server
  • Create all CRM tables, filtered views, etc. in the Name_MSCRM database
  • Update the MSCRM_CONFIG database

If you are a service provider or ISV, you’ll probably want to buy or build some software to automate the provisioning of CRM.  There are a couple vendors who have built this kind of tool specifically for CRM.  There are also several generic ordering/billing/provisioning systems you can buy but you’ll need to do extensive customization of these to get them to work with your CRM implementation as they were not built with CRM in mind.  If you are a “plain-vanilla” CRM hoster, one of these may do the trick.  Unfortunately, if you are an ISV and you want to offer a SaaS option, you’ll probably need to build your own ordering/billing/provisioning system.  This is what we did at CourseMax.  I would have strongly preferred to buy this software but nothing came close to meeting our requirements.  We spent considerable effort building an OBP system as a customization of CRM using custom workflow plug-ins.  While it took a lot of time for us to build the tool, it automates a large part of our business and it is very flexible.  I may post more information about what we did in a future post if there is enough interest.

Understanding CRM’s Database Multitenancy

Each CRM Organization (tenant) has its own database named using the convention of UniqueOrgName_MSCRM.  There is one database named MSCRM_CONFIG that holds mapping data to link a URL that is unique to the organization to the organization’s database.  MSCRM_CONFIG also contains other metadata that pertains to the the CRM implementation.  There is quite a bit more to discuss about CRM’s database multitenancy and the schema of the MSCRM_CONFIG and OrgName_MSCRM databases.  Perhaps I will detail this in a future post.

Misconception #1: “The  MSCRM_CONFIG database holds all of the CRM metadata.”  Not true.  There is metadata in each organization database as well that is specific to that organization.

URLs for Accessing an Organization

On-Premise:

http://CRMSERVERNAME:5555/UniqueOrgName 

Breakdown of the On-Premise URL:

http://
The protocol portion of the URL can be http or https.  Since you are only exposing this traffic to “trusted” networks, it is common to use http (unencrypted).  I fully expect some security professionals to chastise me here for even considering not using https even though it is inside the firewall so I’ll preempt you first.  Every organization has to make their own decision which typically pits greater security against performance and ease of use.

CRMSERVERNAME
The server name portion of the URL can be the “NetBIOS” name of the server as it is shown above or you can use a fully-qualified domain name such as “crmservername.domain.local”.  If you use the NetBIOS name, you’ll need to ensure it can be resolved by all of the PCs on your network using WINS, LMHOSTS, broadcast, or DNS by appending the local or parent paths.  If you use a fully-qualified domain name (FQDN), you’ll need to make sure that all hosts can resolve the name using DNS and you’ll also need to ensure that the FQDN is included in the list of hosts in the Local Intranet zone in the Internet Explorer configuration of every PC on the network.  By default, when the server is identified as being in the “Local Intranet” zone, IE uses automatic Kerberos or NTLM authentication using your domain login credentials.

:5555
TCP port “5555” is not mandatory but it is the default port if you create a new website for CRM during installation.  So, you could also use any of the following URLs:
http://CRMSERVERNAME/UniqueOrgName
http://CRMSERVERNAME:6789/UniqueOrgName

UniqueOrgName
The UniqueOrgName part of the URL must match the unique name you gave the CRM organization when you provisioned it.  Note that the UniqueOrgName is given as a “Virtual Directory” of the CRM application.  This part of the URL is what tells the CRM application which database to use.  Word of warning…be careful when you name an organization.  You cannot easily change this unique organization name after the organization is deployed.  The only way to change it is to restore or reattach the databse then “import” the organization using a different name.

Off-Premise:

https://UniqueOrgName.domain.com

Breakdown of the Off-Premise URL:

https://
While not “technically” mandatory, https (http over SSL encryption) should always be used on untrusted networks.  Your users will be passing credentials over the Internet in clear text if you do not use https.

UniqueOrgName
Again, this part of the name must match the unique name you gave the organization when you provisioned the organization.  Note that the UniqueOrgName is given as a “hostname” for the off-premise URL (as opposed to the use of a “Virtual Directory” for the on-premise URL)

Misconception #2: “You configure CRM’s IFD URL to use a hostname or a virtual directory to refer to the organization.”  False: The IFD URL must be specified as a hostname.  Only the on-premise URL uses a virtual directory to refer to the organization name.

Misconception #3: In the URL “something1.something2.com” something1 is called a subdomain.  I don’t care if this is technically correct or not (somebody can research the W3C spec for me if you like).  “something1” has always been and will always be a hostname to me.  Most of the cheap-o domain registrars and hosters (yes, I use one too) have decided to call everything a subdomain even if it is being used as a hostname.  Perhaps they think this is less confusing to their customers.  I’m not sure why they do it but it doesn’t make sense to me.

domain.com
The domain and TLD (Top-Level Domain e.g. “.com”) portion of the url can be a first-level domain or a subdomain at any level.  However, you can have only one domain or subdomain name for the CRM implementation.  So, you could also use any of the following URLs:
https://UniqueOrgName.subdomain.domain.com
https://UniqueOrgName.subdomain-n.subdomain.domain.com

The wildcard domain certificate must match “*.domain.com” (or “*.subdomain.domain.com” if you are using a subdomain).  See the section on wildcard domain certificates below for more details.

:5443
Yes, I fooled you, there is no port in the above “Off-Premise” URL.  However, you can use a unique TCP port other than 443 and append it to the above URL if you like.  Of course, you will need to configure this port as the SSL port for the web site using IIS Manager in addition to configuring the port in the CRM server configuration.  You’ll need to allow access on your firewall to the CRM server over whatever TCP port you choose here.

Configuring the Internet Facing Deployment (“IFD”) Option 

You can configure IFD either at the time of install using an XML file or after install using the IFD Config tool.  So, do you want to do this the easy way or the hard way?  If you chose the easy way, then go ahead and install CRM then use the IFD Config tool afterwards.  Before the IFD Config tool was released, the only way to configure IFD was to use a crminstall.xml file.  There was not and still is not an option to configure IFD in the setup wizard.  There is also not an option to install individual CRM server roles through the setup wizard.  In the wizard, you have only two choices “Application Server” (roles: AppServer, HelpServer, SDKServer) or “Platform Server” (roles: Async, SDKServer, DeploymentService, DiscoveryService) which are groupings of individual server roles.  You WILL need to use an XML file to install individual roles on a server if the role groupings are not what you want (in many cases they are not and so you might end up doing it the hard way anyway).

Download and read the following document for details on configuring IFD using the IFD Config Tool or the crminstall.xml file:
How to configure an Internet-Facing Deployment for Microsoft Dynamics CRM 4.0

No mater which method you use, you will need to specify the following parameters:

IFD Internal Network Address and Subnet Mask
You can add one or more internal networks to this configuration parameter which are stored in the registry on the CRM Application server.  If the CLIENT’s IP address is found within this network, they will be authenticated automatically (no login form presented…IE just does the authentication automatically) with their domain credentials using Kerberos or NTLM.  If the client’s IP address is not found in this network, they will be presented with the login page.  Keep in mind that the “client” could be any of the following:

  • End-User accessing CRM with Internet Explorer
  • End-User accessing CRM with the CRM Client for Outlook
  • Another Application accessing the CRM Web Services

It is important to note that the client will need to provide the correct URL (see “URLs for Accessing an Organization” above).  The CRM server will not do any automatic redirection. 

A basic understanding of IP Subnetting is helpful in determining what to specify for the IP Address and subnetmask.  Typically, I would recommend that you add all of your internal network addresses that exist behind the firewall.  This will allow you to provide users with a single-sign-on experience.  They will not need to log in to CRM at all.  They will simply access the On-Premise URL and IE will automatically pass the credentials they used when they logged into their PC.  It goes without saying that they should be logging into the corporate domain.  Most organizations use IANA private IP addresses in one of the following ranges:

Network Address: 10.0.0.0 Subnet Mask: 255.0.0.0  (10.0.0.0 to 10.255.255.255)
Network Address: 192.168.0.0 Subnet Mask: 255.255.0.0 (192.168.0.0 to 192.168.255.255)
Network Address: 172.16.0.0 Subnet Mask: 255.240.0.0 (172.16.0.0 to 172.31.255.255)

If you use the 10 network for internal addresses, the easiest thing to do is configure the whole 10 network by using 10.0.0.0 for the network and 255.0.0.0 as the subnet mask. The same would go for 192.168.x.x or 172.16.x.x.

My guess is that this parameter is the source of the most confusion for configuring IFD.  I’m not sure why except that perhaps there is a general lack of knowledge around IP subnetting.  I have read a number of different blog and forum posts where someone has given bad advice regarding this parameter.

Misconception #4: You should always use the IP Address of the Server and 255.255.255.255 for the subnet mask when you configure the IFD Internal Network Address.  False: If you follow this advice, you will force all clients to use IFD unless they are logged in to the console of the CRM server itself.  This would preclude offering single-sign-on to users who access CRM on-premise.  Even if you are running a hosted implementation and all of your USERS are outside of the firewall, you may want to allow for Kerberos authentication for APPLICATIONS that access the CRM web services on the network.

The following three parameters together define the URL that clients will need to specify to access CRM from outside the firewall (Off-Premise):

IFD Domain Scheme
This parameter can be either HTTP or HTTPS.  For security reasons this should always be HTTPS.

IFD App Root Domain
This parameter will be the domain or subdomain that is appended to the unique organization name to form the Off-Premise URL.  For example, if you want clients to use “https://UniqueOrgName.coursemax.com&#8221;
you would use “coursemax.com” for this parameter.

IFD SDK Root Domain
This parameter will typically be the same as the IFD App Root Domain.

The following three parameters together define the URL that clients will need to specify to access CRM from inside the firewall (On-Premise):

AD Domain Scheme
This parameter can be either HTTP or HTTPS

AD App Root Domain
This parameter should be the hostname and port to access the CRM server.  For example, if you want clients to use “http://CRMSERVERNAME:5555/UniqueOrgName&#8221;,
you should specify “CRMSERVERNAME:5555” for this parameter

AD SDK Root Domain
This parameter will typically be the same as the AD App Root Domain.

How to Obtain and Install a Wildcard SSL Certificate for a Multitenant CRM Implementation

While it is possible to create your own CA and generate your own SSL Certificate, you will probably NOT want to do this.  If you do this, you will need to configure the CA as a “Trusted Root CA” in Internet Explorer on every one of your users PCs.  In most situations, this is not practical.  You will probably want to purchase a certificate from a Trusted CA.  Depending on which CA you use, you will pay between $100 and $1,000 (yes there is a wide range of prices, but I won’t get into that here) for a wildcard certificate.

At this point you may be thinking “what is a wildcard SSL certificate and why do I need one.”  If you have ever configured SSL on a web server, you know that you need to acquire and install a certificate which has a “common name” that matches the URL users use to access your web site.  The trouble is that the URL is different for every tenant.  For example, you might have the following Off-Premise Tenant addresses in use:

https://acme-explosives.yourdomain.com
https://northwindtraders.yourdomain.com

https://somethingelse.yourdomain.com

In this case, you would want to purchase a wildcard certificate with a common name of  *.yourdomain.com.  All browsers (we’re only concerned with IE here because that is the only browser Microsoft CRM supports) will accept this SSL certificate as valid for any HOST in the domain yourdomain.com.  Notice that I capitalized the word “host”.  Don’t make the mistake (this nearly cost me some money) in assuming that if you purchase a wildcard SSL certificate with a common name of *.yourdomain.com that it will be valid for SUBDOMAINS within yourdomain.com.  Internet Explorer will NOT accept this certificate if the URL is for a host in a deeper subdomain such as “northwindtraders.crm.yourdomain.com”.  As an aside, Firefox will actually accept a wildcard cert as valid for any host in the domain or a subdomain.  The implementation of this specification varies between browsers.

How to Setup DNS for Multitenant CRM

You’ll need to set up DNS so that your clients can access every organization.  There are two options:

  1. Add an “A” or “CNAME” record every time you provision an organization
  2. Use wildcard DNS

Option 1 is best if you have a small, fixed number of organizations.  You can configure a single A record for your default organization then configure a CNAME record for each organization.  The A record would have a name like “crm.yourdomain.com” and the IP address of your CRM application server (or the VIP if you are using load-balancers).

Option 2 is best for service-providers or ISVs who will be continually provisioning organizations.  You simply configure an A record with the name “*.yourdomain.com” and the IP address of your CRM application server (or the VIP).  I’ve seen another method of doing this documented in a Microsoft knowledgebase article that involves creating a *.yourdomain.com subdomain and configuring a host record for the . address.  However, the prior method works just fine for me so why go to the trouble.  Keep in mind that any URL someone enters that is not specified in another A or CNAME record will resove to your CRM application server or VIP.  So, if someone types “fubar.yourdomain.com” it will resolve to the app server whether you have a fubar organization or not.

The “Default CRM Organization” and “The Mysterious Discovery Service”

I wanted to write something about these subjects but I’m really tired of typing and I think this post is long enough.  In short, use careful consideration when choosing the name of the first CRM organization you provision when installing CRM because it will become the default organization.  The default organization is difficult to impossible to change after the fact (I haven’t found any credible documentation on how to do it that works.  If someone knows how to change it, please share) and in certain situations, this organization will need to contain certain users (another undocumented issue you may run into if you are developing SDK customizations) that you may not want in your customer’s organization.  My advice is to use “CRM” or something generic then don’t plan to use it as a tenant organization. 

The discovery service is just poorly understood and not well-documented in my opinion.  Another blog, another time… 

Relevant Documentation

You’ll need to know where to find all of the documentation you’ll need.  Here’s the list with a summary of what’s included with each document:

Microsoft Dynamics CRM 4.0: Planning and Deployment Guidance for Service Providers
Although the title says “Service Providers” the information is relevant to anyone hosting multi-tenant CRM.  This could be a Service Provider (hosts “vanilla” CRM), an ISV (provides their own customized software built on the CRM platform), or an enterprise hosting multiple instances of the software.  To get this, you’ll download an archive that contains three documents and the “Dynamics CRM Deployment Config Tool”.  The documents are the “Microsoft Dynamics CRM 4.0 Planning Guide for Service Providers“, the “Microsoft Dynamics CRM 4.0 Deployment Walkthrough Guide for Service Providers“, and the “Microsoft Dynamics CRM 4.0 Development Guide for Service Providers”.  As stated in each of these socuments, they “should be considered a supplement to the main product documentation.”  In other words there is an assumption that the reader is familiar with the main product documentation in addition to having knowledge of Windows Server, Active Directory, IIS 6.0, DNS, SQL Server 2005, Exchange Server 2007, and fundamental networking concepts.  However, there is a lot of overlap between the main product documentation and the 319 pages of documentation contained in these three documents.

Microsoft Dynamics CRM 4.0 Planning Guide for Service Providers (PDF, 73 pages)
This is a technical guide that provides a broad range of information for technical professionals who want to host CRM.  Multitenancy is one of the many topics covered.  This is not a short read but it should be on your mandatory reading list.

CRM 4.0 Deployment Walkthrough Guide for Service Providers (PDF, 172 pages)
This is a “how-to” guide with step by step instructions on how to install and configure CRM in various hosting scenarios, including how to enable the Internet-Facing Deployment (IFD) option during the install (it can also be enabled after the install).  The trick in reading this document is to find what is pertinent to your implementation among the 172 pages.  The document goes to great lengths to discuss hosting Exchange in conjunction with CRM, including the side-by-side provisioning of Hosted Messaging and Collaboration (“HMC” which is a solution for SPs to host Exchange and Sharepoint).  If you don’t need to host Exchange, much of this document will not be relevant to you.

How to configure an Internet-Facing Deployment for Microsoft Dynamics CRM 4.0
(12/8/2008, Word Doc, 24 pages)
 This is a brief guide on how to set up CRM so that it will provide a logonform to someone who is accessing the software over the Internet, from outside the firewall.  This is known as forms-based authentication.  This document is an update to the original document released on 1/14/2008 which was an 8 page PDF.  It includes instructions for configuring IFD at install time and using the IFD Configuration tool.  You could also use the info to gain an understanding of how to manually configure IFD by editing the registry, SQL tables, and the web.config file.  Of course, I would NEVER recommend (at least not publicly) editing the SQL tables directly (add all the pertinent warnings about editing the registry or SQL tables directly here).  Seriously though, I would not recommend doing a soup to nuts configuration of IFD manually.  It’s much safer to use the xml configuration file or the IFD Config tool, but it is good to know what to change if you need to do some troubleshooting.  If you have a copy of the original document, delete it and use the new one.  The updated document is much more comprehensive and accurate.

Helpful Posts

Internet Facing Deployment (IFD) Installation Basics (9/18/2008, CRM Team Blog, Shashi Ranjan)
Brief blog post from Shashi Ranjan of the CRM Team summarizing the use of an XML setup file to configure IFD during installation.

Microsoft Dynamics CRM URLs (8/1/2008, CRM Team Blog, Jagan Peri)
Jagan summarizes what URLs to use to access various pages when on-premise or off-premise (outside the firewall)

Microsoft Dynamics CRM 4.0 : Multi-Tenancy with John O’Donnell
Video by John O’Donnell, Microsoft Dynamics ISV Architect Evangelist (no, not the Jimmy Swaggart kind, the Microsoft kind).  John walks through various topics around multitenancy for Microsoft CRM in this 14 minute video.  It’s well worth the time spent.

Lessons learned from using the CRM IFD tool (Customer Effective Blog)
Some helpful tips from our friends at Customer Effective.

Multi-tenancy in CRM 4 (CRM Team Blog, 1/18/2008, Jagin Peri)
A good, brief post on multitenancy in CRM and creating organizations manually or programmatically.

Make Microsoft Dynamics CRM 4.0 client-to-server network communications more secure
Good article on configuring SSL for CRM including a section about how to “Configure Microsoft Dynamics CRM client-to-server communication for Internet-facing deployments.”

I hope this post saves someone countless hours of searching for information and trying to understand CRM 4.0 multitenancy and IFD.  Write a comment and let me know if this post helped please.  Thanks!

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to FurlAdd to Newsvine