How to debug Visual Studio extensions?

Question

I'm currently trying to write my own Visual Studio extension (custom editor). I tried to write some code similarly to the example provided by MS in the SDK, but it does not work.

What happens is, that after running the second, experimental instance of VS and after constructing all necessary classes (editor factory, editor pane and the editor control itself), VS informs, that "Operation cannot be completed" and nothing else happens.

I do have debugger attached, but it does not stop at any errors. Also there is no trace left regarding the extension apart from two traces placed explicitly in my source code:

Entering constructor for: Spooksoft.Spk.SpkPackage
Entering Initialize() of: Spooksoft.Spk.SpkPackage
'devenv.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\Microsoft.VisualStudio.Data.Tools.Delta.UI\v4.0_11.1.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.Data.Tools.Delta.UI.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'devenv.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\Microsoft.TeamFoundation.Controls\v4.0_11.0.0.0__b03f5f7f11d50a3a\Microsoft.TeamFoundation.Controls.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'devenv.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\assembly\GAC_MSIL\Microsoft.TeamFoundation.Client\11.0.0.0__b03f5f7f11d50a3a\Microsoft.TeamFoundation.Client.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.

(...)

'devenv.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\Microsoft.VisualStudio.Web.HTML.Razor.Shims\v4.0_11.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.Web.HTML.Razor.Shims.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'devenv.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\Microsoft.VisualStudio.Web.HTML.Razor.Implementation.Shims.2_0\v4.0_11.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.Web.HTML.Razor.Implementation.Shims.2_0.dll'
The thread 'StatusBar' (0x760) has exited with code 0 (0x0).
The thread '<No Name>' (0xffc) has exited with code 0 (0x0).
The program '[4200] devenv.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).

How may I determine, what went wrong? Where can I search for any error messages?

The source code (if anyone cares).

Answer 1

That may not have resolved my problem, but surely gives more options. With help of MS employees (on MS Forums) I managed to determine, which interfaces are expected to be implemented in VS.

You have to implement ICustomQueryInterface. Its implementation may look like the following:

public CustomQueryInterfaceResult GetInterface(ref Guid iid, out IntPtr ppv) 
{
    Debug.WriteLine(String.Format("Attempt to cast to interface with iid {0}", iid));

    ppv = IntPtr.Zero;
    return CustomQueryInterfaceResult.NotHandled;
}

You'll get the following result:

Attempt to cast to interface with iid f22a0ad5-8f51-4f66-a644-ea64770cf8b7
Attempt to cast to interface with iid 9d71890d-090c-4b67-80c3-4cb55c600b60
Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
Attempt to cast to interface with iid 9d71890d-090c-4b67-80c3-4cb55c600b60
Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
Attempt to cast to interface with iid 9d71890d-090c-4b67-80c3-4cb55c600b60
Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
Attempt to cast to interface with iid 00020400-0000-0000-c000-000000000046
Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
Attempt to cast to interface with iid 8560cecd-dfac-4f7b-9d2a-e6d9810f3443
Attempt to cast to interface with iid 8560cecd-dfac-4f7b-9d2a-e6d9810f3443

This gives us the GUIDs of interfaces, to which VS tries to cast our class. Now we can use the following class to retrieve the actual types from the GUIDs.

As would Seven say, crude, but effective.

public static class Util 
{

    private static Dictionary<string, Type> guidMap;

    static Util()
    {
        guidMap = new Dictionary<string, Type>();

        AppDomain.CurrentDomain.AssemblyLoad += (s, e) =>
        {
            try 
            {
                _ScanAssembly(e.LoadedAssembly);
            } 
            catch 
            {

            }
        };

        foreach (Assembly a in AppDomain.CurrentDomain.GetAssemblies())
        {
            try 
            {
                _ScanAssembly(a);
            } 
            catch 
            {

            }
        }
    }

    private static void _ScanAssembly(Assembly a)
    {
        foreach (Type t in a.GetTypes())
        {
            var attrs = t.GetCustomAttributes<GuidAttribute>(false);
            foreach (var item in attrs) {

                if (!guidMap.ContainsKey(item.Value.ToLower()))
                    guidMap.Add(item.Value.ToLower(), t);
            }
        }
    }

    public static Dictionary<string, Type> GuidMap 
    {
        get 
        {
            return guidMap;
        }
    }
}

Now we can implement ICustomQueryInterface in a little bit different way:

    public CustomQueryInterfaceResult GetInterface(ref Guid iid, out IntPtr ppv) 
    {
        if (Util.GuidMap.ContainsKey(iid.ToString().ToLower())) 
        {
            Type t = Util.GuidMap[iid.ToString()];
            Debug.WriteLine(String.Format("Attempt to cast to interface {0}", t.ToString()));
        }
        else
            Debug.WriteLine(String.Format("Attempt to cast to interface with iid {0}", iid));

        ppv = IntPtr.Zero;
        return CustomQueryInterfaceResult.NotHandled;
    }

And the result is a lot more helpful:

Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsDeferredDocView
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData2
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData2
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData2
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
Attempt to cast to interface System.Runtime.InteropServices.NativeMethods+IDispatch
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
Attempt to cast to interface Microsoft.VisualStudio.TextManager.Interop.SVsCodeWindow
Attempt to cast to interface Microsoft.VisualStudio.TextManager.Interop.SVsCodeWindow

Now I'll try to determine, which of these interfaces are required and which are optional.