Correctly Creating Classes Using xsd.exe

Not a lot of people are familiar with the xsd.exe that ships with Visual Studio. It allows you to create XML/classes from XML/classes. The following are the 4.0 xsd.exe capabilities:

XDR to XSD

Generates an XML schema from an XML-Data-Reduced schema file. XDR is an early XML-based schema format.

XML to XSD

Generates an XML schema from an XML file.

XSD to DataSet

Generates common language runtime DataSet classes from an XSD schema file. The generated classes provide a rich object model for regular XML data.

XSD to Classes

Generates runtime classes from an XSD schema file. The generated classes can be used in conjunction with System.Xml.Serialization.XmlSerializer to read and write XML code that follows the schema.

Classes to XSD

Generates an XML schema from a type or types in a runtime assembly file. The generated schema defines the XML format used by System.Xml.Serialization.XmlSerializer.

It I’m a big fan of using xml schemas (XSDs) to generate classes that can be used to serialize and deserialize objects to and from XML (the XSD to Classes functionality listed above). If your schema changes, just rerunning xsd.exe for the correct schema updates the classes. No manually changes have to be made, including serialization code. It’s a beautiful thing. The problem has been, how do you set up your classes to automatically recompile with changes to the schema, and how do you deal with schemas that import other schemas?

XSDs Importing other XSDs

A common issue that developers of XSDs run into is violating the DRY principle repeatedly with XSD types. For example, let’s say you create a calendar meeting request service that has two XSDs, one for the request XML and one for the response XML. You’ve defined a xs:complexType “Meeting” that includes the date and location:

  <xs:complexType name="Meeting">
    <xs:sequence>
      <xs:element name="Location" type="xs:string"/>
      <xs:element name="Date" type="xs:date"/>
    </xs:sequence>
  </xs:complexType>

But you want to use it in both the request and the response XML. You could just copy and past it into both XSD files, and it will validate just fine, but if you use xsd.exe to generate your classes, it’s going to create two classes of type Meeting, which will cause a compiler error. You could have a separate namespace for each class, but then you’re definitely violating DRY. The answer is to place the Meeting type in a separate XSD and then reference it from both your request and your response XSD. This results in the XSDs below

C:\Solution\Project\Types.xsd
<?xml version="1.0" encoding="utf-8"?>

<xs:schema id="Types"
targetNamespace="http://www.Example.com/Types"
elementFormDefault="qualified"
xmlns="http://www.Example.com/Types"
xmlns:mstns="http://www.Example.com/Types"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
  <xs:complexType name="Meeting">
    <xs:sequence>
      <xs:element name="Location" type="xs:string"/>
      <xs:element name="Date" type="xs:date"/>
    </xs:sequence>
  </xs:complexType>
</xs:schema>

C:\Solution\Project\Request.xsd
<?xml version="1.0" encoding="utf-8" ?>

<xs:schema elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:myTypes="http://www.Example.com/Types">
  <xs:import namespace="http://www.Example.com/Types" schemaLocation="Types.xsd" />
  <xs:element name="Request">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="RqstMeeting" type="myTypes:Meeting"/>
        <xs:element name="RqstName" type="xs:string"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

C:\Solution\Project\Response.xsd

<?xml version="1.0" encoding="utf-8" ?>
<xs:schema elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:myTypes="http://www.Example.com/Types">
  <xs:import namespace="http://www.Example.com/Types" schemaLocation="Types.xsd" />
  <xs:element name="Response">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Accepted" type="xs:boolean"/>
        <xs:element name="AlternateMeeting" type="myTypes:Meeting" minOccurs="0"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

Now we’ve defined our Meeting type in one file, and reused it in both our Request.xsd and Response.xsd.

Getting xsd.exe To Import XSDs

Now that the type has been defined in another file, the xsd.exe will generate this error if you attempt to create the create the Request XML:
C:\Solution\Project>xsd.exe Request.xsd /c

Schema validation warning: Type 'http://www.Example.com/Types:Meeting' is not declared.
Warning: Schema could not be validated. Class generation may fail or may produce incorrect results.
Error: Error generating classes for schema 'C:\Solution\Projects\Request'.

- The datatype 'http://www.Example.com/Types:Meeting' is missing.
If you would like more help, please type "xsd /?".

This is due to the fact that the xsd.exe does not use the schemaLocation hint to find the imported schema. You’ve got to include it as a parameter. in your xsd.exe call:
C:\Solution\Project>xsd.exe Types.xsd Request.xsd /c
This will generate one file call Request.cs that has a Request class, and a Meeting class. Now we just need to create the Response class and we’re good to go. But wait… running “C:\Solution\Project>xsd.exe Types.xsd Response.xsd /c” will create a different file, Response.cs, that contains a Response class and a duplicate Meeting class. Now we’re stuck with another compiler error and no longer DRY.

Getting xsd.exe To Not Create Duplicate Classes

This is a simple fix, but it took me a long time to figure out. You have to use xsd.exe to compile all of your classes at once, so rather than running two separate commands, you just need to run one:
C:\Solution\Project>xsd.exe Types.xsd Request.xsd Response.xsd /c
Now you have one file, Response.xsd, with all three classes in it.

Getting Visual Studio 2010 To Auto Recompile XSD Generated Classes

Using the Project Build Events, you can set the project to always recompile the XSD classes each time you build the project. It is also helpful to rename the file so it isn’t always the name of the last XSD file passed to xsd.exe. Here are the Pre-build event command line values required to auto build the XSD classes and rename the file to XsdGeneratedClasses.cs:

EDIT suggested by Jamie instead of the Registry hack (it worked back in the day for me, don't know if it still does), he suggests using "$(TargetFrameworkSDKToolsDirectory)xsd.exe" to find the path of the xsd.exe. Thanks Jamie!

"$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.0A\@InstallationFolder)\bin\NETFX 4.0 Tools\xsd.exe" "$(ProjectDir)Request.xsd" "$(ProjectDir)Response.xsd" "$(ProjectDir)Types.xsd" /c /o:"$(ProjectDir)"
move "$(ProjectDir)Types.cs" "$(ProjectDir)XsdGeneratedClasses.cs"

Now whenever the project get’s built, the XSD generated classes will always be rewritten by xsd.exe

Extending XSD Generated Classes

Don’t forget that classes created by xsd.exe are all partial classes. It’s a good idea to add default constructors and logic in a separate partial class in a different file. It’s especially helpful for initializing arrays since xsd.exe generated classes use arrays and not ArrayLists or Generic Lists. This allows you to add logic, that won’t be changed when the class is regenerated.

Serializing/Deserializing XSD Generated Classes

Now your code for Serializing and Deserializing your objects is as simple as this:

To Serialize:

XmlSerializer s = new XmlSerializer(typeof(Request));System.IO.TextWriter w = new System.IO.StreamWriter(@"C:\Request.xml");s.Serialize(w, new Request());w.Close();

To Deserialize:

XmlSerializer s = new XmlSerializer(typeof(Request));Request request;System.IO.TextReader r = new System.IO.StreamReader("request.xml");request = (Request)s.Deserialize(r);r.Close();

Performing Asynchronous XML Serialization

If you work at a company anything like mine, you've had to deserialize a large XML file, and been forced to sit there for the Deserialize() method to complete. I haven't done a lot of work with threading, but I figured now was the time to start.

I decided to wrap the normal System.Xml.Serialization.XmlSerializer in a generic class that would encapsulate the threading. I also wanted it to perform some logging if there were issues during deserialization. The process starts with a factory method to return the serializer object.

        public static LoggedXMLSerializer<T> RunDeserializeAsync(string filePath) {
            return new LoggedXMLSerializer<T>(filePath);
        }

Which immediately calls the private constructor:

        private LoggedXMLSerializer(string filePath)
            : this() {
            DeserializingDelegate = PeformDeserialization;

            
            FileSize = new FileInfo(filePath).Length;
            // The XML File tends to take up 110% of space in memory as it does on disc
            FileSize = (long)(FileSize * 1.10);

            TotalMemoryIncrease = 0;
            
            Result = DeserializingDelegate.BeginInvoke(filePath, null, null);
            StartingMemorySize = GetCurrentProcessMemoryInUse();
        }

Which first marks the PerformDeserialization() method as the DeserializationDelegate. It then determines the file size that is being opened to be used later to determine the progress that is left. For now, it only deserializes an actual file, but it could be extended with the other default constructors of the XmlSerializer.

BeginInvoke() is called on the DeserializationDelege, which starts the deserialization and returns an IAsyncResult object. BeginInvoke() starts a new thread, and calls the assigned Delegate. You can then query the IAsyncResult object to see if it finished, or just call EndInvoke() and your primary thread will wait until the secondary thread finishes.

Immediately after the thread is created, the current size of the process is stored to also be used later to determine the progress left.

The PerformDeserialization() method is exactly what you would do if you weren't invoking it on a separate thread. Create an XmlSerialization object,assign an Event Handler for loading issues, open a file with a StreamReader and call Deserialize().

        private T PeformDeserialization(string filePath) {
            XmlSerializer xs = new XmlSerializer(typeof(T));
            xs.UnknownNode += new XmlNodeEventHandler(Xs_UnknownNode);

            StreamReader reader = File.OpenText(filePath);

            return (T)xs.Deserialize(reader);
        }

Since the Deserialize() method is still synchronous, the hardest part has been coming up with a good method of determining the progress of the file load. This is what I came up with, but if you, dear reader, have a better idea, I'd like to hear it.

        /// <summary>
        /// Best guess at progress based on the size of the file, and the amount of increase in the memory of the process
        /// </summary>
        public int Progress {
            get {
                if (Result.IsCompleted) {
                    return 100;
                }
                long currentSize = GetCurrentProcessMemoryInUse();
                if (currentSize < StartingMemorySize + TotalMemoryIncrease) {
                    // For Some reason, the current size of memory is smaller than the starting size plus the increase in memory usage
                    // Assume it is due to some garbage collection in between calls to Progress
                    // Update the starting memory size so it is equal to the current + total increase
                    // This assumes that no additional memory was used to deserialize the XML
                    StartingMemorySize = currentSize - TotalMemoryIncrease;
                } else {
                    TotalMemoryIncrease = currentSize - StartingMemorySize;
                }

                int tempProgress = (int)((currentSize - StartingMemorySize) / (double)FileSize * 100);
                if (tempProgress < 0) {
                    tempProgress = 0;
                }

                if (tempProgress > 125) {
                    // Must have had a bad starting point, move it back to 75%
                    StartingMemorySize = (int)(currentSize - .75 * FileSize);
                    // Reinitialize Total Memory
                    TotalMemoryIncrease = currentSize - StartingMemorySize;
                    tempProgress = 75;
                }

                if (tempProgress > 100) {
                    tempProgress = 99;
                }
                return tempProgress;
            }
    }

The first thing it does, is check the IAsyncResult object to see if it has completed, if it has, then it returns 100%. Done. The next part I added later when I noticed that if I opened up more than one file, the progress of the second file would move to about 25%, then it would drop down to near 0, and stay there until it finished. I'm guessing it is due to the garbage collector collecting a large amount of memory due to the first deserialized object being release. The basic method of determining progress is then calculated, assume that the deserialized XML, will take up nearly the same amount as the serialized, is then performed. Get the increase in memory size since first beginning to deserialize the Xml, and divide it by the size of the file. Then do some checking to see if the progress has grown too large, or is over a 100%. It is not a perfect solution, but was extremely simple to implement, and serves my needs well.

Below is the entire class. Feel free to make comments.

    public class LoggedXMLSerializer<T> {
        private delegate T Deserializer (string path);

        private Deserializer DeserializingDelegate { get; set; }
        private IAsyncResult Result { get; set; }
        private long StartingMemorySize { get; set; }
        private long TotalMemoryIncrease { get; set; }
        private long FileSize { get; set; }
        private Dictionary<string, string> XmlUnknowns {get; set;}

        #region Public Properties

        public T Xml { get; protected set;}

        /// <summary>
        /// Best guess at progress based on the size of the file, and the amount of increase in the memory of the process
        /// </summary>
        public int Progress {
            get {
                if (Result.IsCompleted) {
                    return 100;
                }
                long currentSize = GetCurrentProcessMemoryInUse();
                if (currentSize < StartingMemorySize + TotalMemoryIncrease) {
                    // For Some reason, the current size of memory is smaller than the starting size plus the increase in memory usage
                    // Assume it is due to some garbage collection in between calls to Progress
                    // Update the starting memory size so it is equal to the current + total increase
                    // This assumes that no additional memory was used to deserialize the XML
                    StartingMemorySize = currentSize - TotalMemoryIncrease;
                } else {
                    TotalMemoryIncrease = currentSize - StartingMemorySize;
                }

                int tempProgress = (int)((currentSize - StartingMemorySize) / (double)FileSize * 100);
                if (tempProgress < 0) {
                    tempProgress = 0;
                }

                if (tempProgress > 125) {
                    // Must have had a bad starting point, move it back to 75%
                    StartingMemorySize = (int)(currentSize - .75 * FileSize);
                    // Reinitialize Total Memory
                    TotalMemoryIncrease = currentSize - StartingMemorySize;
                    tempProgress = 75;
                }

                if (tempProgress > 100) {
                    tempProgress = 99;
                }
                return tempProgress;
            }
        }

        /// <summary>
        /// Returns true when the XML has finished being Deserialized
        /// Returns false if it hasn't
        /// </summary>
        public bool IsCompleted {
            get {
                if (Result.IsCompleted) {
                    Xml = DeserializingDelegate.EndInvoke(Result);
                    return true;
                } else {
                    return false;
                }
            }
        }

        #endregion // Public Properties

        /// <summary>
        /// Returns a list of all unknown nodes found in the XML in this format
        /// Entity Name, First Occurance Line Number, First Occurance Line Position
        /// </summary>
        /// <returns></returns>
        public string GetLog() {
            StringBuilder sb = new StringBuilder();
            foreach (var item in XmlUnknowns) {
                sb.Append(item.Key + ", " + item.Value + Environment.NewLine);
            }
            return sb.ToString();
        }


        private LoggedXMLSerializer()
            : base() { // Force Factory Method Use
            XmlUnknowns = new Dictionary<string, string>();
        }

        private LoggedXMLSerializer(string filePath)
            : this() {
            DeserializingDelegate = PeformDeserialization;

            
            FileSize = new FileInfo(filePath).Length;
            // The XML File tends to take up 110% of space in memory as it does on disc
            FileSize = (long)(FileSize * 1.10);

            TotalMemoryIncrease = 0;
            
            Result = DeserializingDelegate.BeginInvoke(filePath, null, null);
            StartingMemorySize = GetCurrentProcessMemoryInUse();
        }

        /// <summary>
        /// Deserializes the XML on a different thread.  Use IsCompleted and Progress to determine status
        /// </summary>
        /// <param name="filePath"></param>
        /// <returns></returns>
        public static LoggedXMLSerializer<T> RunDeserializeAsync(string filePath) {
            return new LoggedXMLSerializer<T>(filePath);
        }

        private T PeformDeserialization(string filePath) {
            XmlSerializer xs = new XmlSerializer(typeof(T));
            xs.UnknownNode += new XmlNodeEventHandler(Xs_UnknownNode);

            StreamReader reader = File.OpenText(filePath);

            return (T)xs.Deserialize(reader);
        }

        private long GetCurrentProcessMemoryInUse() {
            Process process = Process.GetCurrentProcess();
            return process.WorkingSet64;
        }

        private void Xs_UnknownNode(object sender, XmlNodeEventArgs e) {
            if (!XmlUnknowns.ContainsKey(e.Name)) {
                XmlUnknowns.Add(e.Name, e.LineNumber + ", " + e.LinePosition);
            }
        }
    }