Melody's VSTA Blog

Get the latest and greatest information on VSTA here! Find new samples, workarounds, and inside tips in our downloads and blogs. Get your questions answered quickly in our forums or search our site for a FAQ. We're here to help- just ask.

Using a MessageFilter to avoid IDE threading and timing issues.

Due to timing and threading issues with interacting with the IDE programatically, several errors may occur at seemingly random intervals.  These errors include “Application is busy”, “Callee was rejected by caller”, “The operation could not be completed”, or RPC_E_SERVER…. Exceptions.  Because these are ultimately timing issues the best solution is to use a Message Filter to tell the host to wait and try again when a call is rejected.  In the ShapeAppAdvancedCSharp and TestCon samples a simple MessageFilter is implemented which is included below in CSharp and VB.Net.  To implement this class, be sure to update your main method to use the message filter.  For an alternate implementation check out this post: http://msdn2.microsoft.com/en-us/library/ms228772(VS.80).aspx


Visual Basic.Net-
Update main method:

Shared Sub Main()
   Using messageFilter_Value As New MessageFilter()
      'original contents are unchanged
   End Using
End Sub

Message Filter:

Imports System
Imports system.Runtime.InteropServices

' There are several threading issues in the IDE that can cause the STA to get hijacked while
' we're attempting to automate. By default, .Net does not provide a message filter, so
' blocked calls result in RPC_E_SERVER... exceptions. Since this is a timing issue, it would
' be very hard for the host to trap these exceptions and try again. The easiest solution,
' is to implement a message filter which tells the host to wait and try again
' when a call is rejected.
<ComImport()> _
<Guid("00000016-0000-0000-C000-000000000046")> _
<InterfaceType(ComInterfaceType.InterfaceIsIUnknown)> _
Public Interface IMessageFilter
   <PreserveSig()> _
   Function HandleInComingCall(ByVal dwCallType As Integer, _
      ByVal hTaskCaller As IntPtr, _
      ByVal dwTickCount As Integer, _
      ByVal lpInterfaceInfo As IntPtr) As Integer

   <PreserveSig()> _
   Function ReTryRejectedCall(ByVal hTaskCallee As IntPtr, _
      ByVal dwTickCount As Integer, _
      ByVal dwRejectType As Integer) As Integer

   <PreserveSig()> _
   Function MessagePending(ByVal hTaskCallee As IntPtr, _
      ByVal dwTickCount As Integer, _
      ByVal dwPendingType As Integer) As Integer
End Interface

' This needs to be MBRO because winforms will query for this to obtain the component manager
' When it does this, it will call Marshal.GetObjectForIUnknown which will convert this to
' an object and force serialization to type MessageFilter. To avoid this, make it MBRO
' so the cast to com visible types will succeed.
Friend Class MessageFilter
   Inherits MarshalByRefObject
   Implements IDisposable
   Implements IMessageFilter

   Private disposedValue As Boolean = False ' To detect redundant calls
   Private disposed As Boolean
   Private oldFilter As IMessageFilter
   Private Const SERVERCALL_ISHANDLED As Integer = 0
   Private Const PENDINGMSG_WAITNOPROCESS As Integer = 2

   'IDisposable
   Protected Overridable Sub Dispose(ByVal disposing As Boolean)
      If Not Me.disposed Then
         Dim dummy As IMessageFilter
         Dim hr As Integer

         hr = MessageFilter.CoRegisterMessageFilter(Me.oldFilter, dummy)
         System.Diagnostics.Debug.Assert(hr >= 0)
         Me.disposed = True
         System.GC.SuppressFinalize(Me)
      End If
   End Sub

   #Region " IDisposable Support "
   ' This code added by Visual Basic to correctly implement the disposable pattern.
   Public Sub Dispose() Implements IDisposable.Dispose
      ' Do not change this code. Put cleanup code in Dispose(ByVal disposing As Boolean) above.
      Dispose(True)
      GC.SuppressFinalize(Me)
   End Sub
   #End Region

   Public Function HandleInComingCall(ByVal dwCallType As Integer,
                                      ByVal threadIdCaller As System.IntPtr,
  
                                   ByVal dwTickCount As Integer, ByVal lpInterfaceInfo As System.IntPtr)
                                       As Integer Implements IMessageFilter.HandleInComingCall
      'Return the ole default (don't let the call through)
      Return MessageFilter.SERVERCALL_ISHANDLED
   End Function

   Public Function MessagePending(ByVal threadIdCallee As System.IntPtr,
                                  ByVal dwTickCount As Integer,
                                  ByVal dwPendingType As Integer)
                                  As Integer Implements  IMessageFilter.MessagePending
      'perform default processing
      Return MessageFilter.PENDINGMSG_WAITNOPROCESS
   End Function

   Public Function ReTryRejectedCall(ByVal threadIdCallee As System.IntPtr,
                                     ByVal dwTickCount As Integer,
                                     ByVal dwRejectType As Integer)
                                   As Integer Implements IMessageFilter.ReTryRejectedCall
      ' Retry the call in 100 milliseconds
      ' Normally, a host would eventually want to throw a message box after some pre-determined time-out if
      ' the call doesn't get through (like excel does after a few minutes).
      ' "Microsoft Excel is waiting for an ole call to complete"
      ' I leave this as an excecise for the reader.
      Return 100
   End Function

   Friend Sub New()
       Dim hr As Integer

       Me.disposed = False
       hr = MessageFilter.CoRegisterMessageFilter(CType(Me, IMessageFilter), Me.oldFilter)
       System.Diagnostics.Debug.Assert(hr >= 0)
    End Sub

   <DllImport("ole32.dll")> _
   <PreserveSig()> _
   Private Shared Function CoRegisterMessageFilter(ByVal lpMessageFilter As IMessageFilter,
                                                   ByRef lplpMessageFilter As IMessageFilter) As Integer
   End Function
End Class


CSharp-
Update main method:

static void Main()
{
    using (MessageFilter messageFilter = new MessageFilter())
   {
       // original contents are unchanged
   }
}

Message Filter:

//***************************************************************************
//
// Copyright (c) 2006 Microsoft Corporation. All rights reserved.
// This code is licensed under the Visual Studio SDK license terms.
// THIS CODE IS PROVIDED "AS IS" WITHOUT WARRANTY OF
// ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
// IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
// PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
//
//***************************************************************************
using System;
using System.Runtime.InteropServices;

// There are several threading issues in the IDE that can cause the STA to get hijacked while
// we're attempting to automate. By default, .Net does not provide a message filter, so
// blocked calls result in RPC_E_SERVER... exceptions. Since this is a timing issue, it would
// be very hard for the host to trap these exceptions and try again. The easiest solution,
// is to implement a message filter which tells the host to wait and try again
// when a call is rejected.
namespace Microsoft.VisualStudio.Tools.Applications.Samples.ShapeApp
{
   [ComImport()]
   [Guid("00000016-0000-0000-C000-000000000046")]
   [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    public interface IMessageFilter
   {
      [PreserveSig]
      int HandleInComingCall(
   
     int dwCallType,
         IntPtr hTaskCaller,
         int dwTickCount,
         IntPtr lpInterfaceInfo);

      [PreserveSig]
      int RetryRejectedCall(
         IntPtr hTaskCallee,
         int dwTickCount,
         int dwRejectType);

       [PreserveSig]
      int MessagePending(
         IntPtr hTaskCallee,
         int dwTickCount,
         int dwPendingType);
   }

    // This needs to be MBRO because winforms will query for this to obtain the component manager
    // When it does this, it will call Marshal.GetObjectForIUnknown which will convert this to
    // an object and force serialization to type MessageFilter. To avoid this, make it MBRO