Wellington Code Camp – this weekend

It’s been three years since the last (and indeed first) Code Camp in Wellington, so it’s about time we had another one, before the Christchurch guys left us for dust – and now it’s nearly here.  So if you haven’t already signed up, and you’re up for a weekend of software architecture, .NET technology and being cut up in the pit lane by the demon Jeremy Boyd, register here and come along.

The schedule’s on the site, and I reckon it’s going to be an interesting mix of technology how-tos and broader architectural and structural talks.  We’ve unintentionally ended up with several talks that ask people to take a second look at the way they “naturally” do things, which I think will make for plenty of great discussion and hopefully the occasional “aha!” moment.

Finally, since Mash gets disappointed if I write for too long in plain English, I’d just like to mention asynchronous monads.  Thank you and good night.

Possibly the most obscure humour in the world, ever

Via Brian McNamara, at last someone has combined functional programming at its most theoretical with amusing cat photos.  Let’s hear a big “/* You are not expected to understand this */” for… lambdacats!

 

I really need to slap a caption on that photo of Fang reviewing the WPF Property Grid code.

The passive voice strikes back

Geoffrey K Pullum commemorates 50 Years of Stupid Grammar Advice: “What concerns me is that the bias against the passive is being retailed by a pair of authors so grammatically clueless that they don't know what is a passive construction and what isn't… The book’s toxic mix of purism, atavism, and personal eccentricity is… often so misguided that the authors appear not to notice their own egregious flouting of its own rules. They can't help it, because they don't know how to identify what they condemn... English grammar… is much too important to be reduced to a bunch of trivial don’t-do-this prescriptions by a pair of idiosyncratic bumblers who can't even tell when they've broken their own misbegotten rules.”  Too many good bits to quote; well worth a read.

I am kindling. Hear me roar

Cat curled up in front of fireplace = scene of adorable domesticity.

Cat curled up in fireplace = need better cat.

Building Clojure for the CLR

David Miller has announced a port of Clojure, the Lisp dialect for the JVM that’s getting a lot of buzz, to the .NET CLR.  Sources are up at the clojure-contrib site.

It’s pretty easy to get it building and running but if you’re using a recent version of the DLR (I was on 21259) then you’ll probably need to make a couple of minor changes:

• ClojureCLR tries to new up ScriptCode in a few places.  ScriptCode is now an abstract class, so this causes compiler errors.  Changing ScriptCode to LegacyScriptCode seems to fix the problem.  (It probably isn’t the right solution, but it’s enough to get things up and running.)
• If you try to use a .NET API call in the REPL, e.g. (Math/Pow 3 4), you get a runtime error of type ArgumentException complaining about a type error in the printToConsole method.  To work around this, go into Generator.cs (in the Compiler folder), find the Generate(object, bool) method, locate the if (addPrint) body, and add the line finalExpr = Expression.Convert(finalExpr, typeof(object)); at the top of the body of the if-statement.  Again, not sure if this is the right thing to do, but it gets the API calls working.

Things I did not know, number 8073

Plastic bowls can cause mouth acne in cats.

Faster development through less crap

A couple of tips for .NET developers who get frustrated with how slow everything is.

Speeding up XAML editing

To get a much faster and more stable WPF XAML editing experience, right-click a XAML file and choose Open With > Source Code (Text) Editor > Set as Default.  This gets you the full XAML editor, with Intellisense, without bogging you down with the visual designer.  It is about a million times faster, even compared to the designer’s XAML-only view, especially when you have multiple XAML files open.  If you need an excuse to go get some coffee, do your shopping or take a three-month hiking tour of the Himalayas, you still open the visual designer using, wait for it, the View Designer command.  (Thanks to Fabrice Marguerie.)

Speeding up MSDN

To get MSDN online to come up quickly in the browser, switch to the low-bandwidth version.  You don’t get fancy resizing or buckets of graphics, but you get the content and you get it quick.  Click here to see what it’s like and choose “Persist low bandwidth view” if you like it – that way you get the fast view even when following a link to the slow version.  (Thanks to Craig Andera.)

Mathematics and programming

Jeff Atwood disputes the theory that competent programmers should be mathematically inclined.  His conclusion may or may not be correct – I don’t have any evidence one way or the other.  His reasoning, however, is wrong.

Jeff observes that “the vast bulk of code that I've seen consists mostly of the ‘balancing your checkbook’ sort of math, nothing remotely like what you'd find in the average college calculus textbook, even,” and gives the example i = j++ / (x + v) -- “not,” he rightly observes, “exactly the stuff mathletes are made of.”

Right, and nothing to do with being mathematically inclined either.  Jeff seems to be one of those people who believes that what mathematicians do is sit around doing ever harder calculations.  The sort of people who can only imagine Andrew Wiles proved Fermat’s Last Theorem by a fiendishly complicated process of long division.

In reality, the bulk of mathematical effort goes not into performing calculations, but into hypothesising and proving general truths.  Let’s take a quick run down of the kind of skills mathematicians really use in doing this:

• Identifying useful abstractions – separating out the salient points of a problem.
• Formulating a path from A to B – building a high-level plan of attack for a problem or proof.
• Turning that plan of attack into a series of precise and rigorous steps, expressed using an abstract and formal notation.
• Bundling up appropriate partial results into handy lemmas – another example of identifying useful abstractions.
• Thinking about special cases that might invalidate the assumptions of the proof or require special handling.

The core skills of mathematics sound awfully like the core skills of programming.  It is only “very hard … to draw a direct line from ‘good at math’ to ‘good at programming’” if you confuse mathematics with “mathletics.”

Scala for C# programmers, part 6: infix operators

C#, like most object-oriented programming languages, is pretty strict about how you call methods: you use the dot notation, unless the method is a special ‘operator’ method such as operator+, operator== or a conversion operator.  The special operator methods are predefined by the compiler: you can write your own implementation, but you can’t create your own operator names.  You can teach the + operator how to handle your custom type, but you can’t add an exponentiation operator:

int a = b ** c;

C# has three problems with this: first, it doesn’t like the method name “**”; second, it doesn’t like that there’s no . before the name; and third, it doesn’t like that there’s no brackets around the method argument.

To get around the objection to the name, let’s compromise and call it ToThe for now.  So what C# insists on seeing is a.ToThe(b).

Scala, like many functional languages, isn’t so strict.  Scala allows you to use any method with a single argument in an infix position.  Before we can see this in the exponentiation example, we need to pimp the Double type with the toThe method:

import Implicits._
import System.Math._

class DoubleExtensions(d : Double) {
  def toThe(exp: Double): Double = Pow(d, exp)
}

object Implicits {
  implicit def DoubleExtensions(d : Double) = new DoubleExtensions(d)
}

Recall that this is just the Scala idiom for extension methods – it’s the equivalent of writing public static ToThe(this double first, double second) { … } in C#.  (If we were wanting to use infix notation with our own class, we wouldn’t need all this malarkey.)  So now we can write:

val raised = 2.0.toThe(7.0)

Okay, so what do we need to do to get this to work in infix position?  Nothing, it turns out.

val raised = 2.0 toThe 8.0  // it just works

This still doesn’t look much like a built-in operator, but it turns out Scala is less fussy than C# about method names too.

class DoubleExtensions(d : Double) { 
  def **(exp: Double): Double = Pow(d, exp)
}

val raised = 2.0 ** 9.0  // it still just works

Much nicer.

This sorta-kinda works for postfix operators too:

class StringExtensions(s : String) {
  def twice : String = s + s
}

val drivel = "bibble" twice;

The semicolon is required when you invoke a method using postfix notation, which is a bit warty, but life goes on.

Infix and postfix operator support is obviously fairly simple syntactic sugar over normal dot notation.  But this seemingly minor feature is very important in constructing DSLs, allowing Scala to do in internal DSLs what many languages can do only using external tools.  For example, where most languages do parsing via an external file format and a tool to translate that file format into native code (a la lex and yacc), Scala’s parser library makes extensive use of infix and postfix operators to provide a “traditional” syntax for describing a parser, but manages it entirely within the Scala language.  I’ll take a closer look at this in a future post.

Scala for C# programmers, part 5: implicits

It crushes my gentle spirit to call this article “implicits.”  How dreary.  The Scala wiki, describing but one use of implicits, uses the term “pimp my library,” which is so much better.  In the future, all language features will be named in this manner.  Extension methods, for example, will be called “pimp my interface.”  Generics will be called “pimp my type system.”  LINQ will be called “pimp my collections.”  The exceptions will be C++ templates and Lisp macros, which will be referred to using drugs terminology instead.

Terminological difficulties aside, Scala implicits offer some features which will be familiar to the C# programmer, and some which go way beyond pimping and into the aforesaid drugs metaphor.

Let’s start, however, with the pimping.

Extending Types in C# and Scala

Scala, like C#, is statically typed: a class’ methods are compiled into the class definition and are not open for renegotiation.  You cannot, as you might in Ruby or Python, just go ahead and declare additional methods on an existing class.

This is of course very inconvenient.  You end up declaring a load of XxxHelper or XxxUtils classes full of static methods, and having to write verbose calling code such as if (EnumerableUtils.IsEmpty(sequence)) rather than the rather more readable if (sequence.IsEmpty()).

C# 3 addressed this problem by introducing extension methods.  Extension methods are static methods in a XxxHelper or XxxUtils kind of class, except you’re allowed to write them using member syntax.  By defining IsEmpty as an extension method on IEnumerable, you can write if (sequence.IsEmpty()) after all.

Scala disapproves of static classes and global methods, so it plumps for an alternative approach.  You’ll still write a XxxHelper or XxxUtils kind of class, but instead of taking the Xxx to be Helped or Utilised as a method parameter, your class will wrap the Xxx.  Let’s see this in action as we try to add a method to the Double type:

class DoubleExtensions(d : Double) {
  def toThe(exp: Double): Double = System.Math.Pow(d, exp)
}

Notice that toThe is an instance method, and that DoubleExtensions takes a Double as a constructor parameter.  This seems pretty grim, because we’d normally have to access the function like this:

val result = new DoubleExtensions(2.0).toThe(7.0)

Hardly readable.  To make it look nice, Scala requires us to define an implicit conversion from Double to DoubleExtensions:

object Implicits {
  implicit def DoubleExtensions(d : Double) = new DoubleExtensions(d)
}

and to bring that implicit conversion into scope:

import Implicits._

Now we can write this:

val twoToTheSeven = 2.0.toThe(7.0)

and all will be well.  The Double type has apparently been successfully pimped with the toThe method.

This is, of course, just as much an illusion as the C# equivalent.  C# extension methods don’t add methods to a type, and nor do Scala implicit conversions.  What has happened here is that the Scala compiler has looked around for implicit methods that are applicable to the type of 2.0 (namely Double), and return a type that has a toThe method.  Our Implicits.DoubleExtensions method fits the bill, so the Scala compiler silently inserts a call to that method.  At runtime, therefore, Scala calls Implicits.DoubleExtensions(2.0) and calls the toThe of the resulting DoubleExtensions.

If setting this up seems a bit verbose, well, maybe.  C# extension methods are designed to be easily – one might even say implicitly – brought into scope.  That’s very important for operators like the LINQ standard query operators, but it can result in unwanted extension methods being dragged into scope and causing havoc.  Scala requires the caller to be a bit more explicit about implicits, which results in a slightly higher setup cost but gives the caller finer control over which implicit methods are considered.

But as it happens you can avoid the need for separate definitions of Implicits and DoubleExtensions, and get back to a more concise representation by using an anonymous class.  (As you’d expect, Scala anonymous classes are fully capable, like Java ones, rather than the neutered C# version.)  Here’s how it looks:

object Implicits {
  implicit def doubleToThe(d1 : Double) = new {
    def toThe(d2 : Double) : Double = Math.Pow(d1, d2)
  }
}

(Unfortunately, the Scala .NET compiler currently seems to have some problems calling extension methods defined using anonymous classes, so this example only works with the JVM compiler, but you get the idea.)

Well, big deal.  Scala can pimp existing types with new methods just like C#, but using a different syntax.  In related news, Lisp uses a different kind of bracket: film at eleven.  Why should we be interested in Scala implicits if they’re just another take on extension methods?

Implicit Parameters

What we saw above was an implicit method – a method which, like a C# implicit conversion operator, the compiler is allowed to insert a call to without the programmer writing that call.  Scala also has the idea of implicit parameters – that is, parameters which the compiler is allowed to insert a value for without the programmer specifying that value.

That’s just optional parameters with default values, right?  Like C++ and Visual Basic have had since “visual” meant ASCII art on a teletype, and like C# is about to get?  Well, no.

C++, Visual Basic and C# optional parameters have fixed defaults specified by the called function.  For example, if you have a method like this:

public void Fie(int a, int b = 123) { … }

and you call Fie(456), it’s always going to be equivalent to calling Fie(456, 123).

A Scala implicit parameter, on the other hand, gets its value from the calling context.  That allows programmer calling the method to control the implicit parameter value, creating an extensibility point that optional parameters don’t provide.

This probably all sounds a bit weird, so let’s look at an example.  Consider the following Concatenate method:

public T Concatenate<T>(IEnumerable<T> sequence, T seed, Func<T, T, T> concatenator);

We pass this guy a sequence, a start value and a function that combines two values into one, and it returns the result of calling that function across the sequence.  For example, you could pass a sequence of strings, a start value of String.Empty, and (s1, s2) => s1 + s2, and it would return you all the strings concatenated together:

IEnumerable<string> sequence = new string[] { “mog”, “bites”, “man” };
string result = Concatenate(sequence, String.Empty, (s1, s2) => s1 + s2);
// result is “mogbitesman”

But this is a unpleasantly verbose.  We’re having to pass in String.Empty and (s1, s2) => s1 + s2 every time we want to concatenate a sequence of strings.  Not only is this tedious, it also creates the opportunity for error when the boss decides to “help” and passes the literal “String.Empty” as the seed value instead.  (“Oh, and I upgraded all the semi-colons to colons while I was in there.  No, don’t thank me!”)  We’d like to just tell the Concatenate function, “Look, this is how you concatenate strings,” once and for all.

Let’s start out by redefining the Concatenate method in Scala.  I’m going to factor out the seed and the concatenator method into a trait because we’ll typically be defining them together.

trait Concatenator[T] {
  def startValue : T
  def concat(x : T, y : T) : T
}

object implicitParameters {
  def concatenate[T](xs : List[T])(c : Concatenator[T]) : T =
    if (xs.isEmpty) c.startValue
    else c.concat(xs.head, concatenate(xs.tail)(c))
}

We can call this as follows:

object stringConcatenator extends Concatenator[String] {
  def startValue : String = ""
  def concat(x : String, y : String) = x.concat(y)
}

object implicitParameters {
  def main(args: Array[String]) = {
    val result = concatenate(List("mog", "bites", "man"))(stringConcatenator)
    println(result)
  }
}

So far, this looks like the C# version except for the factoring out of the Concatenator trait.  We’re still having to pass in the stringConcatenator at the point of the call.  Let’s fix that:

def concatenate[T](xs : List[T])(implicit c : Concatenator[T]) : T =
  if (xs.isEmpty) c.startValue
  else c.concat(xs.head, concatenate(xs.tail))

We’ve changed two things here.  First, we’ve declared c to be an implicit parameter, meaning the caller can leave it out.  Second, we’ve, er, left it out ourselves, in the recursive call to concatenate(xs.tail).

Well, okay, it’s nice that concatenate now doesn’t have to pass the Concatenator explicitly to the recursive call, but we’re still having to pass in the stringConcatenator object to get things started.  If only there were some way to make the stringConcatenator object itself implicit…

object Implicits {
  implicit object stringConcatenator extends Concatenator[String] {
    def startValue : String = ""
    def concat(x : String, y : String) = x.concat(y)
  }
}

Again, we’ve done two things here.  First, we’ve declared the stringConcatenator object implicit.  Consequently, we’ve had to move it out of the top level, because Scala doesn’t allow implicits at the top level (because they’d pollute the global namespace, being in scope even without an explicit import statement).

Now we can call concatenate like this:

import Implicits._

object implicitParameters {
  def main(args: Array[String]) = {
    val result = concatenate(List("mog", "bites", "man"))
    println(result)
  }
}

And we’ll still get “mogbitesman” as the output.

Let’s review what’s going on here.  The implicit parameter of concatenate has been set to our stringConcatenator, a default value that the concatenate method knew nothing about when it was compiled.  This is somewhere north of what classical optional parameters are capable of, and we’re not finished yet.  Let’s build a listConcatenator.

object Implicits {
  class ListConcatenator[T] extends Concatenator[List[T]] {
    def startValue : List[T] = Nil
    def concat(x : List[T], y : List[T]) = x ::: y
  }
  implicit object stringListConcatenator extends ListConcatenator[String] { }
}

This is a bit vexing.  List in Scala is a generic type, and has a generic concatenation operator called :::.  But we can’t create a generic object, because an object is an instance.  And implicit parameters have to be objects.  So the best we can do is build a generic ListConcatenator class, and then create trivial implicit objects for each generic parameter type we might need.  This seems a bit silly so I am probably missing something here, but this is the best I’ve managed so far.

However, accepting that I am dumb, let’s not worry about the ugly implementation, and see how this is used at the calling end:

val result = concatenate(List(
  List("mog", "bites", "man"),
  List("on", "beard")
))

This displays List(mog, bites, man, on, beard); that is, it concatenates the two List[String]s into one.  Once again, we have not had to pass stringListConcatenator explicitly: the Scala compiler has gone and found it for us.  We can use the exact same calling code to concatenate lists and strings.

Why Should I Care?

Isn’t this pointless?  At the call site, I have access to stringConcatenator and listStringConcatenator.  I can easily pass them in rather than relying on spooky compiler magic to do it for me.  Aren’t implicit parameters just job security for compiler writers?

Yes, implicit parameters are technically unnecessary.  But if we’re going to play that game, C# is technically unnecessary.  You could write all that code in IL.  Extension methods are unnecessary because you could write the static method out longhand.  Optional parameters are unnecessary because you could read the documentation and pass them in explicitly.  Post-It notes are unnecessary because you could fire up Outlook and create a Note instead.

Implicit parameters are about convenience and expressiveness.  Implicit parameters give you a way of describing how a function should handle different situations, without needing to bake those situations into the function logic or to specify them every damn time you call the function.  You don’t want to have to tell the concatenate function whether to use the List or String concatenator every time you call it: the compiler knows what you’re concatenating; specifying how to concatenate it just gives you a chance to get it wrong!

Consequently, implicit parameters – like implicit conversions – contribute to Scala’s ability to support internal DSLs.  By setting up appropriate implicits, you can write code that reads much more naturally than if you had to pepper it with function objects or callbacks.

Conclusion

Scala’s implicit keyword goes somewhat beyond C#’s equivalent.  As in C#, it is used for implicit conversions; unlike C#, this is the idiomatic way to add operations to an existing type, removing the need for the separate extension method syntax.  Implicit parameters have no equivalent in C#.  They are like being able to add default values to a method: just as a C# using statement bring implicit methods into scope, a Scala import statement can bring default values into scope.  If implicit conversions are a way of extending classes, then implicit parameters are a way of extending methods, creating simple, reliable shorthands for complex generic methods, and making up another piece of the Scala DSL jigsaw.

And the big terminological question.  Where do Scala implicits sit on the pimp–drugs scale of metaphors?  Pimping, for now.  But maybe dealing a little on the side, and keeping an eye on the main chance.