[ACCEPTED]-Best practice for debug Asserts during Unit testing-assert

This is a perfectly valid question.

First 63 of all, many people are suggesting that 62 you are using assertions wrongly. I think 61 many debugging experts would disagree. Although 60 it is good practice to check invariants 59 with assertions, assertions shouldn't be 58 limited to state invariants. In fact, many 57 expert debuggers will tell you to assert 56 any conditions that may cause an exception 55 in addition to checking invariants.

For example, consider 54 the following code:

if (param1 == null)
    throw new ArgumentNullException("param1");

That's fine. But when 53 the exception is thrown, the stack gets 52 unwound until something handles the exception 51 (probably some top level default handler). If 50 execution pauses at that point (you may 49 have a modal exception dialog in a Windows 48 app), you have the chance to attach a debugger, but 47 you have probably lost a lot of the information 46 that could have helped you to fix the issue, because 45 most of the stack has been unwound.

Now consider 44 the following:

if (param1 == null)
{
    Debug.Fail("param1 == null");
    throw new ArgumentNullException("param1");
}

Now if the problem occurs, the 43 modal assert dialog pops up. Execution is 42 paused instantaneously. You are free to 41 attach your chosen debugger and investigate 40 exactly what's on the stack and all the 39 state of the system at the exact point of 38 failure. In a release build, you still get 37 an exception.

Now how do we handle your unit 36 tests?

Consider a unit test that tests the 35 code above that includes the assertion. You 34 want to check that the exception is thrown 33 when param1 is null. You expect that particular 32 assertion to fail, but any other assertion 31 failures would indicate that something is 30 wrong. You want to allow particular assertion 29 failures for particular tests.

The way you 28 solve this will depend upon what languages 27 etc. you're using. However, I have some 26 suggestions if you are using .NET (I haven't 25 actually tried this, but I will in the future 24 and update the post):

1. Check Trace.Listeners. Find any instance of DefaultTraceListener and set AssertUiEnabled to false. This stops the modal dialog from popping up. You could also clear the listeners collection, but you'll get no tracing whatsoever.
2. Write your own TraceListener which records assertions. How you record assertions is up to you. Recording the failure message may not be good enough, so you may want to walk the stack to find the method the assertion came from and record that too.
3. Once a test ends, check that the only assertion failures that occured were the ones you were expecting. If any others occurred, fail the test.

For an example of a 23 TraceListener that contains the code to 22 do a stack walk like that, I'd search for 21 SUPERASSERT.NET's SuperAssertListener and 20 check its code. (It's also worth integrating 19 SUPERASSERT.NET if you're really serious 18 about debugging using assertions).

Most unit 17 test frameworks support test setup/teardown 16 methods. You may want to add code to reset 15 the trace listener and to assert that there 14 aren't any unexpected assertion failures 13 in those areas to mimimize duplication and 12 prevent mistakes.

UPDATE:

Here is an example 11 TraceListener that can be used to unit test 10 assertions. You should add an instance to 9 the Trace.Listeners collection. You'll probably 8 also want to provide some easy way that 7 your tests can get hold of the listener.

NOTE: This 6 owes an awful lot to John Robbins' SUPERASSERT.NET.

/// <summary>
/// TraceListener used for trapping assertion failures during unit tests.
/// </summary>
public class DebugAssertUnitTestTraceListener : DefaultTraceListener
{
    /// <summary>
    /// Defines an assertion by the method it failed in and the messages it
    /// provided.
    /// </summary>
    public class Assertion
    {
        /// <summary>
        /// Gets the message provided by the assertion.
        /// </summary>
        public String Message { get; private set; }

        /// <summary>
        /// Gets the detailed message provided by the assertion.
        /// </summary>
        public String DetailedMessage { get; private set; }

        /// <summary>
        /// Gets the name of the method the assertion failed in.
        /// </summary>
        public String MethodName { get; private set; }

        /// <summary>
        /// Creates a new Assertion definition.
        /// </summary>
        /// <param name="message"></param>
        /// <param name="detailedMessage"></param>
        /// <param name="methodName"></param>
        public Assertion(String message, String detailedMessage, String methodName)
        {
            if (methodName == null)
            {
                throw new ArgumentNullException("methodName");
            }

            Message = message;
            DetailedMessage = detailedMessage;
            MethodName = methodName;
        }

        /// <summary>
        /// Gets a string representation of this instance.
        /// </summary>
        /// <returns></returns>
        public override string ToString()
        {
            return String.Format("Message: {0}{1}Detail: {2}{1}Method: {3}{1}",
                Message ?? "<No Message>",
                Environment.NewLine,
                DetailedMessage ?? "<No Detail>",
                MethodName);
        }

        /// <summary>
        /// Tests this object and another object for equality.
        /// </summary>
        /// <param name="obj"></param>
        /// <returns></returns>
        public override bool Equals(object obj)
        {
            var other = obj as Assertion;
            
            if (other == null)
            {
                return false;
            }

            return
                this.Message == other.Message &&
                this.DetailedMessage == other.DetailedMessage &&
                this.MethodName == other.MethodName;
        }

        /// <summary>
        /// Gets a hash code for this instance.
        /// Calculated as recommended at http://msdn.microsoft.com/en-us/library/system.object.gethashcode.aspx
        /// </summary>
        /// <returns></returns>
        public override int GetHashCode()
        {
            return
                MethodName.GetHashCode() ^
                (DetailedMessage == null ? 0 : DetailedMessage.GetHashCode()) ^
                (Message == null ? 0 : Message.GetHashCode());
        }
    }

    /// <summary>
    /// Records the assertions that failed.
    /// </summary>
    private readonly List<Assertion> assertionFailures;

    /// <summary>
    /// Gets the assertions that failed since the last call to Clear().
    /// </summary>
    public ReadOnlyCollection<Assertion> AssertionFailures { get { return new ReadOnlyCollection<Assertion>(assertionFailures); } }

    /// <summary>
    /// Gets the assertions that are allowed to fail.
    /// </summary>
    public List<Assertion> AllowedFailures { get; private set; }

    /// <summary>
    /// Creates a new instance of this trace listener with the default name
    /// DebugAssertUnitTestTraceListener.
    /// </summary>
    public DebugAssertUnitTestTraceListener() : this("DebugAssertUnitTestListener") { }

    /// <summary>
    /// Creates a new instance of this trace listener with the specified name.
    /// </summary>
    /// <param name="name"></param>
    public DebugAssertUnitTestTraceListener(String name) : base()
    {
        AssertUiEnabled = false;
        Name = name;
        AllowedFailures = new List<Assertion>();
        assertionFailures = new List<Assertion>();
    }

    /// <summary>
    /// Records assertion failures.
    /// </summary>
    /// <param name="message"></param>
    /// <param name="detailMessage"></param>
    public override void Fail(string message, string detailMessage)
    {
        var failure = new Assertion(message, detailMessage, GetAssertionMethodName());

        if (!AllowedFailures.Contains(failure))
        {
            assertionFailures.Add(failure);
        }
    }

    /// <summary>
    /// Records assertion failures.
    /// </summary>
    /// <param name="message"></param>
    public override void Fail(string message)
    {
        Fail(message, null);
    }

    /// <summary>
    /// Gets rid of any assertions that have been recorded.
    /// </summary>
    public void ClearAssertions()
    {
        assertionFailures.Clear();
    }

    /// <summary>
    /// Gets the full name of the method that causes the assertion failure.
    /// 
    /// Credit goes to John Robbins of Wintellect for the code in this method,
    /// which was taken from his excellent SuperAssertTraceListener.
    /// </summary>
    /// <returns></returns>
    private String GetAssertionMethodName()
    {
        
        StackTrace stk = new StackTrace();
        int i = 0;
        for (; i < stk.FrameCount; i++)
        {
            StackFrame frame = stk.GetFrame(i);
            MethodBase method = frame.GetMethod();
            if (null != method)
            {
                if(method.ReflectedType.ToString().Equals("System.Diagnostics.Debug"))
                {
                    if (method.Name.Equals("Assert") || method.Name.Equals("Fail"))
                    {
                        i++;
                        break;
                    }
                }
            }
        }

        // Now walk the stack but only get the real parts.
        stk = new StackTrace(i, true);

        // Get the fully qualified name of the method that made the assertion.
        StackFrame hitFrame = stk.GetFrame(0);
        StringBuilder sbKey = new StringBuilder();
        sbKey.AppendFormat("{0}.{1}",
                             hitFrame.GetMethod().ReflectedType.FullName,
                             hitFrame.GetMethod().Name);
        return sbKey.ToString();
    }
}

You 5 can add Assertions to the AllowedFailures 4 collection at the start of each test for 3 the assertions you expect.

At the end of 2 every test (hopefully your unit testing 1 framework supports a test teardown method) do:

if (DebugAssertListener.AssertionFailures.Count > 0)
{
    // TODO: Create a message for the failure.
    DebugAssertListener.ClearAssertions();
    DebugAssertListener.AllowedFailures.Clear();
    // TODO: Fail the test using the message created above.
}

IMHO debug.asserts rock. This great article shows how 4 to stop them from interrupting your unit 3 test by adding an app.config to your unit 2 testing project, and disabling the dialog 1 box:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.diagnostics>
    <assert assertuienabled="false"/>
</system.diagnostics>

As others have mentioned, Debug asserts 24 are meant for things that should always be true. (The fancy term 23 for this is invariants).

If your unit test is passing 22 in bogus data that is tripping the assert, then 21 you have to ask yourself the question - why 20 is that happening?

• If the function under test is supposed to deal with the bogus data, then clearly that assert shouldn't be there.
• If the function is not equipped to deal with that kind of data (as indicated by the assert), then why are you unit testing for it?

The second point is one 19 that quite a few developers seem to fall 18 into. Unit test the heck out of all the 17 things your code is built to deal with, and 16 Assert or throw exceptions for everything 15 else - After all, if your code is NOT build 14 to deal with those situations, and you cause 13 them to happen, what do you expect to happen?
You 12 know those parts of the C/C++ documentation 11 that talk about "undefined behaviour"? This 10 is it. Bail and bail hard.

Update to clarify: The 9 flipside of this is that you end up realising 8 you should only use Debug.Assert for internal things 7 calling other internal things. If your code 6 is exposed to 3rd parties (i.e. it's a library 5 or something) then there is no limit to 4 what input you can expect, and thus you 3 should validate properly and throw exceptions 2 or whatever, and you should unit test for 1 that too

Assertions in your code are (ought to be) statements 13 to the reader that say "this condition should 12 always be true at this point." Done with 11 some discipline, they can be part of ensuring 10 that the code is correct; most people use 9 them as debug print statements. Unit Tests 8 are code that demonstrates that your code correctly 7 performs a particular test case; don'e well, they 6 can both document the reuirements, and raise 5 your confidence that the code is indeed 4 correct.

Get the difference? The program 3 assertions help you make it correct, the 2 unit tests help you develop someone else's 1 confidence that the code is correct.

A good unit test setup will have the ability 15 to catch asserts. If an assert is triggered 14 the current test should fail and the next 13 is run.

In our libraries low-level debug 12 functionality such as TTY/ASSERTS have handlers 11 that are called. The default handler will 10 printf/break, but client code can install 9 custom handlers for different behavior.

Our 8 UnitTest framework installs its own handlers 7 that log messages and throw exceptions on 6 asserts. The UnitTest code will then catch 5 these exceptions if they occur and log them 4 as an fail, along with the asserted statement.

You 3 can also include assert testing in your 2 unit test - e.g.

CHECK_ASSERT(someList.getAt(someList.size() + 1); // test 1 passes if an assert occurs

I took the approach of disabling the assert 7 only where needed as opposed to doing it 6 project wide. Here is an approach where 5 the assert can be suspend so it doesn't 4 interfere with the test flow.

public static class TraceListenerCollectionEx
{
    /// <summary>
    /// This is a helper class that allows us to suspend asserts / all trace listeners
    /// </summary>
    public class SuspendTrackerDisposable : IDisposable
    {
        private readonly TraceListenerCollection _traceListenerCollection;
        private readonly TraceListener[] _suspendedListeners;

        public SuspendTrackerDisposable(TraceListenerCollection traceListenerCollection)
        {
            _traceListenerCollection = traceListenerCollection;

            var numListeners = traceListenerCollection.Count;
            _suspendedListeners = new TraceListener[numListeners];
            for( int index = 0; index < numListeners; index += 1 )
                _suspendedListeners[index] = traceListenerCollection[index];

            traceListenerCollection.Clear();
        }

        public void Dispose()
        {
            _traceListenerCollection.AddRange(_suspendedListeners);
        }
    }

    public static SuspendTrackerDisposable AssertSuspend(this TraceListenerCollection traceListenerCollection) => new SuspendTrackerDisposable(traceListenerCollection);
}

Here is an 3 example usage within a test:

    [TestMethod]
    public void EnumDefaultTest()
    {
        using(Trace.Listeners.AssertSuspend()) {
            Enum<CarClass>.DefaultValue.ShouldBe(CarClass.Unknown);  
        }
    }

The code executed 2 within the using block, only one line in 1 this case, will have their asserts disabled.

Do you mean C++/Java asserts for "programming 15 by contract" assertions, or CppUnit/JUnit 14 asserts? That last question leads me to 13 believe that it's the former.

Interesting 12 question, because it's my understanding 11 that those asserts are often turned off 10 at runtime when you deploy to production. (Kinda 9 defeats the purpose, but that's another 8 question.)

I'd say that they should be left 7 in your code when you test it. You write 6 tests to ensure that the pre-conditions 5 are being enforced properly. The test should 4 be a "black box"; you should be acting as 3 a client to the class when you test. If 2 you happen to turn them off in production, it 1 doesn't invalidate the tests.

First to have both Design by Contract assertions 19 and unit tests, your unit testing framework 18 shall be able to catch the assertions. If 17 your unit tests abort because of a DbC abort, then 16 you simply cannot run them. The alternative 15 here is to disable those assertions while 14 running (read compiling) your unit tests.

Since 13 you're testing non-public functions, what 12 is the risk of having a function invoked 11 with invalid argument ? Don't your unit 10 tests cover that risk ? If you write your 9 code following the TDD (Test-Driven Development) technique, they 8 should.

If you really want/need those Dbc-type 7 asserts in your code, then you can remove 6 the unit tests that pass the invalid arguments 5 to the methods having those asserts.

However, Dbc-type 4 asserts can be useful in lower level functions 3 (that is not directly invoked by the unit 2 tests) when you have coarse-grained unit 1 tests.

You should keep your debug asserts, even 26 with unit tests in place.

The problem here 25 is not differentiating between Errors and 24 Problems.

If a function checks its arguments 23 which are erroneous, it should not result 22 in a debug assertion. Instead it should 21 return an error value back. It was an Error 20 to call the function with wrong parameters.

If 19 a function is passed correct data, but cannot 18 operate properly due to the runtime has 17 run out of memory, then the code should 16 issue a debug assert due to this Problem. That 15 an example of fundamental assumptions which 14 if they don't hold, "all bets are off", so 13 you must terminate.

In your case, do write 12 the unit test that supplies erroneous values 11 as arguments. It should expect an error 10 return value (or similar). Getting an assert? -- refactor 9 the code to produce an error instead.

Note 8 a bug-free problem can still trigger asserts; e.g. the 7 hardware could break. In your question, you 6 mentioned integration testing; indeed, asserting 5 against incorrectly composed integrated 4 systems is assert territory; e.g. incompatible 3 library version loaded.

Note, the reason 2 for "debug"-asserts is a trade-off between 1 being diligent/safe and being fast/small.

Like the others have mentioned, the Debug.Assert statements 11 should always be true, even if arguements are 10 incorrect, the assertion should be true 9 to stop the app getting into an invalid 8 state etc.

Debug.Assert(_counter == somethingElse, "Erk! Out of wack!");

You should not be able to test 7 this (and probably don't want to because 6 there is nothing you can do really!)

I could 5 be way off but I get the impression that 4 perhaps the asserts you may be talking about 3 are better suited as "argument exceptions", e.g.

if (param1 == null)
  throw new ArgumentNullException("param1", "message to user")

That 2 kind of "assertion" in your code is still 1 very testable.

PK :-)

It has been a while since this question 8 has been asked, but I think I have a different 7 way of verifying Debug.Assert() calls from 6 within a unit test using C# code. Note the 5 #if DEBUG ... #endif block, which is needed for skipping the 4 test when not running in debug configuration 3 (in which case, Debug.Assert() won't be 2 fired anyway).

[TestClass]
[ExcludeFromCodeCoverage]
public class Test
{
    #region Variables              |

    private UnitTestTraceListener _traceListener;
    private TraceListenerCollection _originalTraceListeners;

    #endregion

    #region TestInitialize         |

    [TestInitialize]
    public void TestInitialize() {
        // Save and clear original trace listeners, add custom unit test trace listener.
        _traceListener = new UnitTestTraceListener();
        _originalTraceListeners = Trace.Listeners;
        Trace.Listeners.Clear();
        Trace.Listeners.Add(_traceListener);

        // ... Further test setup
    }

    #endregion
    #region TestCleanup            |

    [TestCleanup]
    public void TestCleanup() {
        Trace.Listeners.Clear();
        Trace.Listeners.AddRange(_originalTraceListeners);
    }

    #endregion

    [TestMethod]
    public void TheTestItself() {
        // Arrange
        // ...

        // Act
        // ...
        Debug.Assert(false, "Assert failed");



    // Assert

#if DEBUG        
    // NOTE This syntax comes with using the FluentAssertions NuGet package.
    _traceListener.GetWriteLines().Should().HaveCount(1).And.Contain("Fail: Assert failed");
#endif

    }
}

The UnitTestTraceListener 1 class looks like follows:

[ExcludeFromCodeCoverage]
public class UnitTestTraceListener : TraceListener
{
    private readonly List<string> _writes = new List<string>();
    private readonly List<string> _writeLines = new List<string>();

    // Override methods
    public override void Write(string message)
    {
        _writes.Add(message);
    }

    public override void WriteLine(string message)
    {
        _writeLines.Add(message);
    }

    // Public methods
    public IEnumerable<string> GetWrites()
    {
        return _writes.AsReadOnly();
    }

    public IEnumerable<string> GetWriteLines()
    {
        return _writeLines.AsReadOnly();
    }

    public void Clear()
    {
        _writes.Clear();
        _writeLines.Clear();
    }
}

Does heavy use of unit tests discourage 82 the use of debug asserts?

No. The opposite. Unit 81 testing makes Debug asserts much more valuable, by 80 double-checking the internal state while 79 running the white-box tests you've written. Enabling 78 Debug.Assert during unit test is essential, because 77 you rarely ship DEBUG-enabled code (unless 76 performance is not important at all). The 75 only two times DEBUG code is run is when 74 you are either 1) doing that tiny bit of 73 integration testing you really do, all good 72 intentions aside, and 2) running unit tests.

It 71 is easy to instrument code with Debug.Assert 70 tests to check invariants as you write it. These 69 checks serve as sanity checks when unit 68 tests run.

The other things Assert does is 67 point to exactly the first point in the 66 code where things went wrong. This can greatly 65 reduce debugging time when your unit test 64 does find the problem.

This increases the value 63 of unit tests.

It seems like a debug assert 62 firing in the code under test implies the 61 unit test shouldn't exist or the debug assert 60 shouldn't exist.

Case in point. This question 59 is about a real thing that happens. Right? Therefore 58 you need debug asserts in your code, and 57 you need them to trigger during unit tests. The 56 possibility that a debug assert might fire 55 during a unit test clearly demonstrates 54 that debug asserts should be enabled during 53 unit tests.

An assert firing means that either 52 your tests are using your internal code 51 incorrectly (and should be fixed), or some 50 of the code under test is calling other 49 internal code incorrectly, or that somewhere 48 a fundamental assumption is wrong. You do 47 not write tests because you think your assumptions 46 are wrong, you... actually, you do. You 45 write tests because at least some of your 44 assumptions are probably wrong. Redundancy 43 is OK in this situation.

"There can be only 42 one" seems like a reasonable principle. Is 41 this the common practice? Or do you disable 40 your debug asserts when unit testing, so 39 they can be around for integration testing?

Redundancy 38 hurts nothing but the runtime of your unit 37 tests. If you really have 100% coverage, runtime 36 might be an issue. Otherwise, no I strongly 35 disagree. There is nothing wrong with checking 34 your assumption automatically in the middle 33 of a test. That's practically the definition 32 of "testing".

Also here is an example that 31 I believe shows the dilema: A unit test 30 passes invalid inputs for a protected function 29 that asserts it's inputs are valid. Should 28 the unit test not exist? It's not a public 27 function. Perhaps checking the inputs would 26 kill perf? Or should the assert not exist? The 25 function is protected not private so it 24 should be checking it's inputs for safety.

Usually, it 23 is not the purpose of a unit test framework 22 is to test the behavior of your code when 21 the invariant assumptions have been violated. In 20 other words, if the documentation you wrote 19 says "if you pass null as a parameter, results 18 are undefined", you do not need to verify 17 that results are indeed unpredictable. If 16 the failure results are clearly defined, they 15 are not undefined, and 1) it should not 14 be a Debug.Assert, 2) you should define 13 exactly what the results are, and 3) test 12 for that result. If you need to unit test 11 the quality of your internal debug assertions, then 10 1) Andrew Grant's approach of making Assertion 9 frameworks a testable asset should probably 8 be checked as the answer, and 2) wow you 7 have awesome test coverage! And I think 6 this is largely a personal decision based 5 on the project requirements. But I still 4 think debug asserts are essential and valuable.

In 3 other words: Debug.Assert() greatly increases 2 the value of unit tests, and the redundancy 1 is a feature.