Scala for C# programmers, part 5: implicits

It crushes my gentle spirit to call this article “implicits.”  How dreary.  The Scala wiki, describing but one use of implicits, uses the term “pimp my library,” which is so much better.  In the future, all language features will be named in this manner.  Extension methods, for example, will be called “pimp my interface.”  Generics will be called “pimp my type system.”  LINQ will be called “pimp my collections.”  The exceptions will be C++ templates and Lisp macros, which will be referred to using drugs terminology instead.

Terminological difficulties aside, Scala implicits offer some features which will be familiar to the C# programmer, and some which go way beyond pimping and into the aforesaid drugs metaphor.

Let’s start, however, with the pimping.

Extending Types in C# and Scala

Scala, like C#, is statically typed: a class’ methods are compiled into the class definition and are not open for renegotiation.  You cannot, as you might in Ruby or Python, just go ahead and declare additional methods on an existing class.

This is of course very inconvenient.  You end up declaring a load of XxxHelper or XxxUtils classes full of static methods, and having to write verbose calling code such as if (EnumerableUtils.IsEmpty(sequence)) rather than the rather more readable if (sequence.IsEmpty()).

C# 3 addressed this problem by introducing extension methods.  Extension methods are static methods in a XxxHelper or XxxUtils kind of class, except you’re allowed to write them using member syntax.  By defining IsEmpty as an extension method on IEnumerable, you can write if (sequence.IsEmpty()) after all.

Scala disapproves of static classes and global methods, so it plumps for an alternative approach.  You’ll still write a XxxHelper or XxxUtils kind of class, but instead of taking the Xxx to be Helped or Utilised as a method parameter, your class will wrap the Xxx.  Let’s see this in action as we try to add a method to the Double type:

class DoubleExtensions(d : Double) {
  def toThe(exp: Double): Double = System.Math.Pow(d, exp)
}

Notice that toThe is an instance method, and that DoubleExtensions takes a Double as a constructor parameter.  This seems pretty grim, because we’d normally have to access the function like this:

val result = new DoubleExtensions(2.0).toThe(7.0)

Hardly readable.  To make it look nice, Scala requires us to define an implicit conversion from Double to DoubleExtensions:

object Implicits {
  implicit def DoubleExtensions(d : Double) = new DoubleExtensions(d)
}

and to bring that implicit conversion into scope:

import Implicits._

Now we can write this:

val twoToTheSeven = 2.0.toThe(7.0)

and all will be well.  The Double type has apparently been successfully pimped with the toThe method.

This is, of course, just as much an illusion as the C# equivalent.  C# extension methods don’t add methods to a type, and nor do Scala implicit conversions.  What has happened here is that the Scala compiler has looked around for implicit methods that are applicable to the type of 2.0 (namely Double), and return a type that has a toThe method.  Our Implicits.DoubleExtensions method fits the bill, so the Scala compiler silently inserts a call to that method.  At runtime, therefore, Scala calls Implicits.DoubleExtensions(2.0) and calls the toThe of the resulting DoubleExtensions.

If setting this up seems a bit verbose, well, maybe.  C# extension methods are designed to be easily – one might even say implicitly – brought into scope.  That’s very important for operators like the LINQ standard query operators, but it can result in unwanted extension methods being dragged into scope and causing havoc.  Scala requires the caller to be a bit more explicit about implicits, which results in a slightly higher setup cost but gives the caller finer control over which implicit methods are considered.

But as it happens you can avoid the need for separate definitions of Implicits and DoubleExtensions, and get back to a more concise representation by using an anonymous class.  (As you’d expect, Scala anonymous classes are fully capable, like Java ones, rather than the neutered C# version.)  Here’s how it looks:

object Implicits {
  implicit def doubleToThe(d1 : Double) = new {
    def toThe(d2 : Double) : Double = Math.Pow(d1, d2)
  }
}

(Unfortunately, the Scala .NET compiler currently seems to have some problems calling extension methods defined using anonymous classes, so this example only works with the JVM compiler, but you get the idea.)

Well, big deal.  Scala can pimp existing types with new methods just like C#, but using a different syntax.  In related news, Lisp uses a different kind of bracket: film at eleven.  Why should we be interested in Scala implicits if they’re just another take on extension methods?

Implicit Parameters

What we saw above was an implicit method – a method which, like a C# implicit conversion operator, the compiler is allowed to insert a call to without the programmer writing that call.  Scala also has the idea of implicit parameters – that is, parameters which the compiler is allowed to insert a value for without the programmer specifying that value.

That’s just optional parameters with default values, right?  Like C++ and Visual Basic have had since “visual” meant ASCII art on a teletype, and like C# is about to get?  Well, no.

C++, Visual Basic and C# optional parameters have fixed defaults specified by the called function.  For example, if you have a method like this:

public void Fie(int a, int b = 123) { … }

and you call Fie(456), it’s always going to be equivalent to calling Fie(456, 123).

A Scala implicit parameter, on the other hand, gets its value from the calling context.  That allows programmer calling the method to control the implicit parameter value, creating an extensibility point that optional parameters don’t provide.

This probably all sounds a bit weird, so let’s look at an example.  Consider the following Concatenate method:

public T Concatenate<T>(IEnumerable<T> sequence, T seed, Func<T, T, T> concatenator);

We pass this guy a sequence, a start value and a function that combines two values into one, and it returns the result of calling that function across the sequence.  For example, you could pass a sequence of strings, a start value of String.Empty, and (s1, s2) => s1 + s2, and it would return you all the strings concatenated together:

IEnumerable<string> sequence = new string[] { “mog”, “bites”, “man” };
string result = Concatenate(sequence, String.Empty, (s1, s2) => s1 + s2);
// result is “mogbitesman”

But this is a unpleasantly verbose.  We’re having to pass in String.Empty and (s1, s2) => s1 + s2 every time we want to concatenate a sequence of strings.  Not only is this tedious, it also creates the opportunity for error when the boss decides to “help” and passes the literal “String.Empty” as the seed value instead.  (“Oh, and I upgraded all the semi-colons to colons while I was in there.  No, don’t thank me!”)  We’d like to just tell the Concatenate function, “Look, this is how you concatenate strings,” once and for all.

Let’s start out by redefining the Concatenate method in Scala.  I’m going to factor out the seed and the concatenator method into a trait because we’ll typically be defining them together.

trait Concatenator[T] {
  def startValue : T
  def concat(x : T, y : T) : T
}

object implicitParameters {
  def concatenate[T](xs : List[T])(c : Concatenator[T]) : T =
    if (xs.isEmpty) c.startValue
    else c.concat(xs.head, concatenate(xs.tail)(c))
}

We can call this as follows:

object stringConcatenator extends Concatenator[String] {
  def startValue : String = ""
  def concat(x : String, y : String) = x.concat(y)
}

object implicitParameters {
  def main(args: Array[String]) = {
    val result = concatenate(List("mog", "bites", "man"))(stringConcatenator)
    println(result)
  }
}

So far, this looks like the C# version except for the factoring out of the Concatenator trait.  We’re still having to pass in the stringConcatenator at the point of the call.  Let’s fix that:

def concatenate[T](xs : List[T])(implicit c : Concatenator[T]) : T =
  if (xs.isEmpty) c.startValue
  else c.concat(xs.head, concatenate(xs.tail))

We’ve changed two things here.  First, we’ve declared c to be an implicit parameter, meaning the caller can leave it out.  Second, we’ve, er, left it out ourselves, in the recursive call to concatenate(xs.tail).

Well, okay, it’s nice that concatenate now doesn’t have to pass the Concatenator explicitly to the recursive call, but we’re still having to pass in the stringConcatenator object to get things started.  If only there were some way to make the stringConcatenator object itself implicit…

object Implicits {
  implicit object stringConcatenator extends Concatenator[String] {
    def startValue : String = ""
    def concat(x : String, y : String) = x.concat(y)
  }
}

Again, we’ve done two things here.  First, we’ve declared the stringConcatenator object implicit.  Consequently, we’ve had to move it out of the top level, because Scala doesn’t allow implicits at the top level (because they’d pollute the global namespace, being in scope even without an explicit import statement).

Now we can call concatenate like this:

import Implicits._

object implicitParameters {
  def main(args: Array[String]) = {
    val result = concatenate(List("mog", "bites", "man"))
    println(result)
  }
}

And we’ll still get “mogbitesman” as the output.

Let’s review what’s going on here.  The implicit parameter of concatenate has been set to our stringConcatenator, a default value that the concatenate method knew nothing about when it was compiled.  This is somewhere north of what classical optional parameters are capable of, and we’re not finished yet.  Let’s build a listConcatenator.

object Implicits {
  class ListConcatenator[T] extends Concatenator[List[T]] {
    def startValue : List[T] = Nil
    def concat(x : List[T], y : List[T]) = x ::: y
  }
  implicit object stringListConcatenator extends ListConcatenator[String] { }
}

This is a bit vexing.  List in Scala is a generic type, and has a generic concatenation operator called :::.  But we can’t create a generic object, because an object is an instance.  And implicit parameters have to be objects.  So the best we can do is build a generic ListConcatenator class, and then create trivial implicit objects for each generic parameter type we might need.  This seems a bit silly so I am probably missing something here, but this is the best I’ve managed so far.

However, accepting that I am dumb, let’s not worry about the ugly implementation, and see how this is used at the calling end:

val result = concatenate(List(
  List("mog", "bites", "man"),
  List("on", "beard")
))

This displays List(mog, bites, man, on, beard); that is, it concatenates the two List[String]s into one.  Once again, we have not had to pass stringListConcatenator explicitly: the Scala compiler has gone and found it for us.  We can use the exact same calling code to concatenate lists and strings.

Why Should I Care?

Isn’t this pointless?  At the call site, I have access to stringConcatenator and listStringConcatenator.  I can easily pass them in rather than relying on spooky compiler magic to do it for me.  Aren’t implicit parameters just job security for compiler writers?

Yes, implicit parameters are technically unnecessary.  But if we’re going to play that game, C# is technically unnecessary.  You could write all that code in IL.  Extension methods are unnecessary because you could write the static method out longhand.  Optional parameters are unnecessary because you could read the documentation and pass them in explicitly.  Post-It notes are unnecessary because you could fire up Outlook and create a Note instead.

Implicit parameters are about convenience and expressiveness.  Implicit parameters give you a way of describing how a function should handle different situations, without needing to bake those situations into the function logic or to specify them every damn time you call the function.  You don’t want to have to tell the concatenate function whether to use the List or String concatenator every time you call it: the compiler knows what you’re concatenating; specifying how to concatenate it just gives you a chance to get it wrong!

Consequently, implicit parameters – like implicit conversions – contribute to Scala’s ability to support internal DSLs.  By setting up appropriate implicits, you can write code that reads much more naturally than if you had to pepper it with function objects or callbacks.

Conclusion

Scala’s implicit keyword goes somewhat beyond C#’s equivalent.  As in C#, it is used for implicit conversions; unlike C#, this is the idiomatic way to add operations to an existing type, removing the need for the separate extension method syntax.  Implicit parameters have no equivalent in C#.  They are like being able to add default values to a method: just as a C# using statement bring implicit methods into scope, a Scala import statement can bring default values into scope.  If implicit conversions are a way of extending classes, then implicit parameters are a way of extending methods, creating simple, reliable shorthands for complex generic methods, and making up another piece of the Scala DSL jigsaw.

And the big terminological question.  Where do Scala implicits sit on the pimp–drugs scale of metaphors?  Pimping, for now.  But maybe dealing a little on the side, and keeping an eye on the main chance.