Wolf's recent King of Xcode post reminded me of one of my major gripes with Xcode. As of Xcode 2.1, Apple's developer tools have shipped with support for writing unit tests in Objective-C, C++, and C. I'm a huge fan of unit testing, so this was exciting news, when it first hit the streets. For Objective-C, they incorporated OCUnit. OCUnit is a mature, well-tested, and Open Source ( BSD-style) unit testing framework. It's been around nearly as long as JUnit, and it shows. It's pretty darned good.

However, for C++, they used a framework called CPlusTest. What's that? Never heard of it? That's because Apple created it. Apparently none of the multitude of existing frameworks were good enough. So they created their own. And made it closed source.

So surely, if you were going to create your own, it would be better than all existing frameworks? I mean why go through all the trouble just to come up short? Well, unfortunately, I don't find CPlusTest to be very good at all. I've used CppUnit in another project, and I'd use CppUnit over CPlusTest, hands down. I'll review CPlusTest using the same seven criteria outlined in Noel Llopis' article, because why mess with a good thing.

  1. Minimal amount of work needed to add new tests. It is very poor in this area. Apart from adding the test methods to the header and the implementation files, you need to manually create a global variable for each and every test method. From Apple's documentation: 

    MyTests test1(TEST_INVOCATION(MyTests, MyFirstTest));
MyTests test2(TEST_INVOCATION(MyTests, MySecondTest));

    So not only do you need to type the class name twice for every method, you have to come up with a unique name for the global variable. test1 and test2 are fine for a small example, but it quickly becomes unwieldy and prone to collisions. What if you have hundreds or thousands, of test methods? This manual setup is crying out for a macro. Sure, it's not too hard to create one yourself, but this is basic functionality that should be provided by the framework.

  2. Easy to modify and port. It gets failing marks here, since it is closed source. Heaven forbid you're writing portable code that needs to run on another platform. You're SOL with CPlusTest. Oh, and CPlusTest requires gcc 4.0. Trying to run a test application compiled with gcc 3.3 gives one of these errors for every test method you run: 

    testrunner(1141) malloc: ***  Deallocation of a pointer not malloced:  
0x1800608; This could be a double free(), or free() called with the  
middle of an allocated block; Try setting environment variable  
MallocHelp to see tools to help debug

    And if you waste your time trying to track down this error because you think it's in your own code (like I did), you'll find it's somewhere in TestCase::~TestCase(). Oh, and did I mention that it's closed source? So, don't even think about trying to fix this error. The only solution is to only use gcc 4.0 with CPlusTest.

  3. Supports fixtures. Yes, it does support the standard setUp() and tearDown() methods. However, it does not get the bonus points as all objects need to be created dynamically, rather than on the stack.

  4. Handles exceptions and crashes well. Again, it scores failing marks here. If a test method throws an exception, the only output you get is: 

    failed: Unknown failure.

    I can't even begin to explain just how unbelievably useless this is. Here's an exception that derives from std::exception. The very least it should do is print out the message returned from what(). Contrast this with the output from the standard gcc exception handler, if an exception bubbles out past main(): 

    terminate called after throwing an instance of 'MyException'
  what():  My exception happened

    Brilliant! It gives the message and the name of the class! Why can't CPlusTest do this?

  5. Good assert functionality. There's one, lone assert statement: CPTAssert(). And it just checks for true/false. So to check for equality, you have to do: CPTAssert(5 == object.someMethod()). Because of this, a failing assert doesn't tell you the actual returned value. Most other frameworks have an assertEqual() variant that will give a message like "Expected 5, but got 3," which is quite helpful. And the better frameworks have dozens of assert statements. Again, failing marks.

  6. Supports different outputs. I think so, but this isn't that big of a deal for me. The fact that it prints output to stderr and Xcode can parse it is good enough for me.

  7. Supports suites. I don't think so, but again, this doesn't really matter to me.

So overall CPlusTest is quite poor. I'd give it a 2 out of 10. As a contrast, I'd give CppUnit a 7 out of 10. The only advantage CPlusTest has is that it comes bundled with Xcode and works as an "Independent Target". If I were to start a new project today, though, I would stay as far away from CPlusTest as possible. I'd easily give up the Independent Target functionality for decent assert statements and error handling, even if it meant compiling and installing my own framework.

Note: I've only used the Xcode 2.2 variant of CPlusTest. It's quite possible that Xcode 2.3 fixed some of these issues. However, CPlusTest isn't mentioned at all in the release notes, so I'm not optimistic.