Improved Method To Avoid A System.InvalidOperationException Multi-Threaded Race Condition In CRM 2011

In order to improve performance creating connections to CRM via the C# SDK, we followed the steps outlined here and here, to cache the ServiceManagement and Credentials objects.  After some quick testing, everything ran smoothly and we pushed the changes to prod.  A couple days later, these exceptions started showing up in our logs:

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at System.Collections.Generic.List`1.Enumerator.MoveNext()
   at Microsoft.Xrm.Sdk.Client.ServiceConfiguration`1.CreateLocalChannelFactory()

A quick Google search turned up this page, which listed the problem being caused by a race condition when creating a connection to CRM, and enabling the proxy types on the OrganizationServiceProxy.  I decided to look into what the SDK was actually doing to create the race condition and this is what I found in the call site where the exception was being thrown:

private ChannelFactory<TService> CreateLocalChannelFactory()
{      
     lock (ServiceConfiguration<TService>._lockObject)
      {
          ServiceEndpoint local_0 = new ServiceEndpoint(this.CurrentServiceEndpoint.Contract, this.CurrentServiceEndpoint.Binding, this.CurrentServiceEndpoint.Address);
          foreach (IEndpointBehavior item_0 in (Collection<IEndpointBehavior>)this.CurrentServiceEndpoint.Behaviors)
              local_0.Behaviors.Add(item_0);
          local_0.IsSystemEndpoint = this.CurrentServiceEndpoint.IsSystemEndpoint;
          local_0.ListenUri = this.CurrentServiceEndpoint.ListenUri;
          local_0.ListenUriMode = this.CurrentServiceEndpoint.ListenUriMode;
          local_0.Name = this.CurrentServiceEndpoint.Name;
          ChannelFactory<TService> local_2 = new ChannelFactory<TService>(local_0);
          if (this.ClaimsEnabledService || this.AuthenticationType == AuthenticationProviderType.LiveId)
              ChannelFactoryOperations.ConfigureChannelFactory<TService>(local_2);
          local_2.Credentials.IssuedToken.CacheIssuedTokens = true;
          return local_2;
      }
}

Even with the Lock statement, the CurrentServiceEndpoind.Behaviors call apparently was still throwing the exception. I then checked the OrganizationServiceProxy to see what EnableProxyTypes() was doing:

public void EnableProxyTypes(Assembly assembly)
{
      ClientExceptionHelper.ThrowIfNull((object) assembly, "assembly");
      ClientExceptionHelper.ThrowIfNull((object) this.ServiceConfiguration, "ServiceConfiguration");
      OrganizationServiceConfiguration serviceConfiguration = this.ServiceConfiguration as OrganizationServiceConfiguration;
      ClientExceptionHelper.ThrowIfNull((object) serviceConfiguration, "orgConfig");
      serviceConfiguration.EnableProxyTypes(assembly);
}

Hmm, nothing there looked to be updating the Behaviors collection. Must be something in the OrganizationServiceConfiguration’s EnableProxyType():

public void EnableProxyTypes(Assembly assembly)
{
      ClientExceptionHelper.ThrowIfNull((object) assembly, "assembly");
      ClientExceptionHelper.ThrowIfNull((object) this.CurrentServiceEndpoint, "CurrentServiceEndpoint");
      lock (this._lockObject)
      {
          ProxyTypesBehavior local_0 = this.CurrentServiceEndpoint.Behaviors.Find<ProxyTypesBehavior>();
          if (local_0 != null)
              ((Collection<IEndpointBehavior>) this.CurrentServiceEndpoint.Behaviors).Remove((IEndpointBehavior) local_0);
          this.CurrentServiceEndpoint.Behaviors.Add((IEndpointBehavior) new ProxyTypesBehavior(assembly));
      }
}

And there it was, the cause of the race condition. One thread calls CreateLocalChannelFactory() while another thread calls EnableProxyTypes(). Even though they are different OrganizationsServiceProxies, they share the same ServiceConfiguration. Even though they are both wrapped in lock statements, they are using different lock objects.

The fix suggested by the only Google result for this error is to add a check to see if the ServiceConfiguration’s CurrentServiceEndpoint has any EndpointBehaviors before enabling the proxy types. This still potentially (although very unlikely) still allows for the exception to occur. I decided to simplify it. Since the only thing the OrganizationServiceProxy does in it’s EnableProxyTypes() is pass the call onto the ServiceConfiguration’s EnableProxyTypes(), and since the ServiceConfiguration is shared amongst all threads, the call to EnableProxyTypes() can be performed directly after the creation of the ServiceConfiguration, before it is returned and used by the OrganizationServiceProxy. This removes the race condition as well as having to check for any existing behaviors before calling EnableProxyTypes(). Below is the utility class that we use to create our OrganizationServiceProxies, sharing the ServiceConfiguration and Credentials.

Before diving into the code, here are some things that aren’t shown below:

• CrmServiceEntity is a class that contains all of the information required to connect to CRM. It overrides Equals(), so it is a valid Key to use.
• GetOrAddSafe is an extension method that ensures that only once CrmServiceCreationInfo object gets created per CrmServiceEntity.

And now the code, which is pretty simple. A call comes in to CreateService, with a CrmServiceEntity parameter. GetOrAddSafe is then called, looking for any existing value in the ConcurrentDictionary. If none is found, it calls the constructor for CrmServiceCreationInfo with the CrmServiceEntity parameter. Keep in mind this is locked and will not be called twice for the same CrmServiceEntity.

The ServiceConfiguration and ClientCredential get created as normal, but then if Proxy Types are enabled, the EnableProxyTypes is called on the Service Configuration. There is some reflection magic required to call the method since it is an internal class, as well as a fail safe incase Microsoft ever changes the class name. But the end result is before the ServiceConfiguration object is ever returned, it should have it’s proxy settings set, which means the race condition will never happen. Enjoy!

private static ConcurrentDictionary<CrmServiceEntity, CrmServiceCreationInfo> _crmServiceCreationInfos = new ConcurrentDictionary<CrmServiceEntity, CrmServiceCreationInfo>();
private static readonly object _crmServiceCreationLock = new object();
private static OrganizationServiceProxy CreateService(CrmServiceEntity entity)
{
      var crmServiceCreationInfo = _crmServiceCreationInfos.GetOrAddSafe(_crmServiceCreationLock, entity, e => new CrmServiceCreationInfo(e));
      var orgService = new OrganizationServiceProxy(crmServiceCreationInfo.ServiceConfiguration, crmServiceCreationInfo.ClientCredential);
      if (entity.ImpersonationUserId != Guid.Empty)
      {
          orgService.CallerId = entity.ImpersonationUserId;
      }
      return orgService;
}


private class CrmServiceCreationInfo{
      public IServiceManagement<IOrganizationService> ServiceConfiguration { get; set; }
      public ClientCredentials ClientCredential { get; set; }
      public CrmServiceCreationInfo(CrmServiceEntity entity)
      {
          var orgUri = GetOrganizationServiceUri(entity);
          ServiceConfiguration = ServiceConfigurationFactory.CreateManagement<IOrganizationService>(orgUri);
          ClientCredential = GetCredentials(entity);
          if (entity.EnableProxyTypes)
          {
              // As of at least CRM 2011 Rollup 15 there exists the potential that sharing the Service Configuration and EnablingProxyTypes could cause a 
              // System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
              //    at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
              //    at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
              //    at System.Collections.Generic.List`1.Enumerator.MoveNext()
              //    at Microsoft.Xrm.Sdk.Client.ServiceConfiguration`1.CreateLocalChannelFactory()
              // http://social.microsoft.com/Forums/en-US/d8d81294-5c11-4490-824d-649c653c7335/linq-exception-occurs-while-retrieving-paged-crm-data-in-a-multihreaded-manner

              // Rather than not enabling the proxy types if it has already been enabled which could still cause the issue,
              // enable it here which is guaranteed to only execute once.
              // type should be of type OrganizationServiceConfiguration which is an internal type.  If something changes
              // Create a temporary OrganizationServiceProxy to then fix the issue.

              var type = ServiceConfiguration.GetType();
              var method = type.GetMethod("EnableProxyTypes", new[] { typeof(System.Reflection.Assembly) });
              if (method == null)
              {
                  LogManager.GetCurrentClassLogger().Warn("EnableProxyTypes doesn't exist for " + type.FullName);
                  using (var orgService = new OrganizationServiceProxy(ServiceConfiguration, ClientCredential))
                  {
                      orgService.EnableProxyTypes(GetEarlyBoundProxyAssembly());
                  }
              }
              else
              {
                  method.Invoke(ServiceConfiguration, new Object[] { GetEarlyBoundProxyAssembly() });
              }
          }
      }
}

ASP.Net Control to Display Session

I wanted to be able to see all my Session data on my webpage, but only when I actually had a debugger attached.  My solution was a two step approach:

1. Create a user control that ouputs the session into a table
2. Add that user control to the to the page if a debugger is attached

To accomplish step 1, I created a user control which is located below.  It overrides the Render method, directly writing a text html table to the HtmlTextWriter.  It contains two other methods, one to create the table to display the session information in, iterating over all the values in the session, and another that uses reflection to output the value of all properties of non-simple types.  

public class SessionDisplayControl : UserControl

 public SessionDisplayControl()
 {
 }

 protected override void Render(HtmlTextWriter writer)
 {
     base.Render(writer);
     writer.Write(GetOutputState());
 }

 protected string GetOutputState()
 {
     System.Text.StringBuilder sb = new System.Text.StringBuilder(2000);
     sb.AppendLine("<table border='1' align='center'><tr><td colspan='2'>There are " + Session.Contents.Count + " Session variables</td></tr>");
     foreach (string name in Session.Contents)
     {
         System.Collections.IEnumerable enumeratable = Session[name] as System.Collections.IEnumerable;
         if (enumeratable == null || Session[name].GetType() == typeof(String))
         {
             sb.AppendLine(string.Format(@"<tr><td>{0}</td><td>{1}</td></tr>", HttpUtility.HtmlEncode(name), GetObjectValue(Session[name] ?? "<NULL>")));
         }
         else
            {
             sb.AppendLine(@"<tr><td>" + HttpUtility.HtmlEncode(name) + @"</td><td><table border='1'>");
             int i = 0;
             foreach (object o in enumeratable)
             {
                 sb.AppendLine(string.Format("<tr><td>Item({0})</td>", i++));
                 sb.AppendLine(string.Format(@"<td>{0}</td></tr>", GetObjectValue(o)));
             }
             sb.AppendLine(@"</table></tr>");
         }
     }
     sb.AppendLine("</table>");
     return sb.ToString();
 }

 private static string GetObjectValue(Object obj)
 {
     if (obj is string || obj is bool || obj is int || obj is long || obj is double || obj is decimal || obj is DateTime)
     {
         return HttpUtility.HtmlEncode(obj.ToString());
     }

     System.Text.StringBuilder sb = new System.Text.StringBuilder(500);
     System.Reflection.PropertyInfo[] properties = obj.GetType().GetProperties();
     foreach (System.Reflection.PropertyInfo property in properties)
     {
         try
            {
             sb.Append(HttpUtility.HtmlEncode(property.Name));
             sb.Append(": ");
             sb.Append(HttpUtility.HtmlEncode((property.GetValue(obj, null) ?? "<NULL>").ToString()));
             sb.Append("<br/>");
         }
         catch (Exception ex)
         {
             sb.Append("ERROR: " + HttpUtility.HtmlEncode(ex.Message) + "<br/>");
         }
     }
     return sb.ToString();
 }
}

To accomplish step 2, all that was needed was to override the OnLoad event in my BasePage class from which all other classes in my website inherit, check for a debugger, and add my user control from step 1 to the page if one was attached.

    protected override void  OnLoad(EventArgs e)
   {
    base.OnLoad(e);
       if (System.Diagnostics.Debugger.IsAttached && !IsPostBack)
       {
           this.Controls.Add(new SessionDisplayControl());
       }
   }

Now whenever I have a debugger attached my session will be displayed at the bottom of the page.  This has helped me clean up unused session variables numerous times.  Enjoy!

Automated CSLA Testing Helper

I was trying to come up with my first test project using the Tester built into VS 2008. I wanted to be able to do some basic CRUD testing on my Csla Business Objects.  I quickly realized that I didn't want to have to go through and set all of my properties by hand for every single class.  Even if I had all the time in the world to do that, if they changed the database schema, which would result in me having to re-code-gen the BOs, I would have to edit my test scripts to handle any new columns (which I would always forget) and remove any columns that no longer exist.  What a pain.

Enter Reflection.  I was able to create a generic SetPropertyValues function that loops through all the properties of any BusinessObject and automatically assigns all them all a different value.  

This is how it works:

    1. Using reflection, I loop through all the properties on the BO with a counter that is incremented each time

    2. I look at the type of the property and assign it a value based off of its type, using the counter as a base

    3. I then save the value off to a generic list of PropertyInfo, and string values to compare with later.

Thats it.

I then wraped that funciton with two other functions, one to perform the intial insert, and one with a different seed value, to perform an update test.  Below is the .Net test class I setup for my customer class, and the SetPropertyValues function that is used to simplify the test.

Enjoy.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Reflection;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace BusinessObjectsUnitTest {
    static class G2TestHelper {

        public static List<PropertyCallerValuePair> SetInitialPropertyValues<T>(Csla.BusinessBase<T> target) where T : Csla.BusinessBase<T> {
            return SetPropertyValues(target, 0);
        }

        public static List<PropertyCallerValuePair> SetUpdatedPropertyValues<T>(Csla.BusinessBase<T> target) where T : Csla.BusinessBase<T> {
            return SetPropertyValues(target, 1);
        }

        private static List<PropertyCallerValuePair> SetPropertyValues<T>(Csla.BusinessBase<T> target, int intialValue) where T : Csla.BusinessBase<T> {
                        List<PropertyCallerValuePair> values = new List<PropertyCallerValuePair>();
            string stringValue = string.Empty;

            foreach (PropertyInfo property in target.GetType().GetProperties(BindingFlags.Instance | BindingFlags.Public | BindingFlags.DeclaredOnly).Where(p => p.CanWrite && !(p.PropertyType is Csla.Core.IBusinessObject))) {

                if(typeof(string).IsAssignableFrom(property.PropertyType)){
                    stringValue = intialValue.ToString();
                    property.SetValue(target, stringValue, null);
                } else if (typeof(bool).IsAssignableFrom(property.PropertyType)) {
                } else if (typeof(long).IsAssignableFrom(property.PropertyType) ||
                           typeof(int).IsAssignableFrom(property.PropertyType) ||
                           typeof(decimal).IsAssignableFrom(property.PropertyType)) {
                    stringValue = intialValue.ToString();
                    property.SetValue(target, intialValue, null);
                } else if (typeof(Csla.SmartDate).IsAssignableFrom(property.PropertyType)) {
                    Csla.SmartDate time = new Csla.SmartDate(DateTime.Now.AddDays(intialValue));
                    stringValue = time.ToString();
                    property.SetValue(target, time, null);
                }else if (typeof(DateTime).IsAssignableFrom(property.PropertyType)){
                    DateTime time = DateTime.Now.AddDays(intialValue);
                    stringValue = time.ToString();
                    property.SetValue(target, stringValue, null);
                } else {
                    throw new Exception("Unrecognized Type");
                }
                
                values.Add(new PropertyCallerValuePair(property, stringValue));
                intialValue++;
            }

            return values;
        }

        public static void AssertAreEqual<T>(Csla.BusinessBase<T> target, List<PropertyCallerValuePair> values) where T : Csla.BusinessBase<T> {
            foreach (PropertyCallerValuePair property in values) {
                Assert.AreEqual(property.Value, property.Property.GetValue(target, null).ToString());
            }
        }
    }

    public class PropertyCallerValuePair
        public PropertyInfo Property { get; set; }
        public string Value { get; set; }

        public PropertyCallerValuePair(PropertyInfo property, string value) {
            Property = property;
            Value = value;
        }
    }
}

Hear is the extremely simple test script would could easily be code generated:

        /// <summary>
        ///A test for Customer CRUD
        ///</summary>
        [TestMethod()]
        public void CustomerCRUD() {
            //Begin Create
            Customer expected = Customer.NewCustomer();
            List<PropertyCallerValuePair> values = G2TestHelper.SetInitialPropertyValues(expected);
            Customer actual = expected.Save();
            //Test Create
            G2TestHelper.AssertAreEqual(actual, values);
            //End Create

            expected = actual;

            //Begin Read
            actual = Customer.GetCustomer(expected.CustId);
            //Test Read
            G2TestHelper.AssertAreEqual(actual, values);
            //End Read

            expected = actual;

            //Begin Update
            values = G2TestHelper.SetUpdatedPropertyValues(expected);
            expected = expected.Save();
            G2TestHelper.AssertAreEqual(expected, values);
            actual = Customer.GetCustomer(expected.CustId);
            //Test Update
            G2TestHelper.AssertAreEqual(actual, values);
            //End Update

            expected = actual;

            //Begin Delete
            Customer.DeleteCustomer(expected.CustId);
            try {
                actual = Customer.GetCustomer(expected.CustId);
            } catch {
                actual = null; 
            }
            //Test Delete
            Assert.IsNull(actual);
            //End Delete
        }

Updating A Private Automatic Property

I had an issue at work today where I was trying to update a public get, private set automatic propety using reflection.

My first issue was I didn't now that when you reflect a type, you can access all the inherited public scoped methods and properties and fields, but you can't access any inherited private methods, properties, and fields. Once I had that figured out, I just had to figure out what the name of the field was.

I downloaded Red Gate's .Net Reflector (an awesome tool) and dissambled my sample test program. I then looked at the property generated by the compiler,

// Fields
[CompilerGenerated]
private long <CustId>k__BackingField;

// Properties
public long CustId

    [CompilerGenerated]
    get
    {
return this.<CustId>k__BackingField;
    }
    private [CompilerGenerated]
    set
    {
        this.<CustId>k__BackingField = value;
    }

And as you can quickly see, the name of the backing field for an automatic property is as follows, k__BackingField. Once I had the backing name figured out, I could update just like any other nonpublic field:

WinPart parent = new WinPart(_mainForm);
parent.GetType().GetField("<CustId>k__BackingField", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance).SetValue(this, newCustId);