« February 2009 | Main | April 2009 »

March 29, 2009

Scala for C# programmers, part 6: infix operators

C#, like most object-oriented programming languages, is pretty strict about how you call methods: you use the dot notation, unless the method is a special ‘operator’ method such as operator+, operator== or a conversion operator.  The special operator methods are predefined by the compiler: you can write your own implementation, but you can’t create your own operator names.  You can teach the + operator how to handle your custom type, but you can’t add an exponentiation operator:

int a = b ** c;

C# has three problems with this: first, it doesn’t like the method name “**”; second, it doesn’t like that there’s no . before the name; and third, it doesn’t like that there’s no brackets around the method argument.

To get around the objection to the name, let’s compromise and call it ToThe for now.  So what C# insists on seeing is a.ToThe(b).

Scala, like many functional languages, isn’t so strict.  Scala allows you to use any method with a single argument in an infix position.  Before we can see this in the exponentiation example, we need to pimp the Double type with the toThe method:

import Implicits._
import System.Math._

class DoubleExtensions(d : Double) {
  def toThe(exp: Double): Double = Pow(d, exp)

object Implicits {
  implicit def DoubleExtensions(d : Double) = new DoubleExtensions(d)

Recall that this is just the Scala idiom for extension methods – it’s the equivalent of writing public static ToThe(this double first, double second) { … } in C#.  (If we were wanting to use infix notation with our own class, we wouldn’t need all this malarkey.)  So now we can write:

val raised = 2.0.toThe(7.0)

Okay, so what do we need to do to get this to work in infix position?  Nothing, it turns out.

val raised = 2.0 toThe 8.0  // it just works

This still doesn’t look much like a built-in operator, but it turns out Scala is less fussy than C# about method names too.

class DoubleExtensions(d : Double) { 
  def **(exp: Double): Double = Pow(d, exp)

val raised = 2.0 ** 9.0  // it still just works

Much nicer.

This sorta-kinda works for postfix operators too:

class StringExtensions(s : String) {
  def twice : String = s + s

val drivel = "bibble" twice;

The semicolon is required when you invoke a method using postfix notation, which is a bit warty, but life goes on.

Infix and postfix operator support is obviously fairly simple syntactic sugar over normal dot notation.  But this seemingly minor feature is very important in constructing DSLs, allowing Scala to do in internal DSLs what many languages can do only using external tools.  For example, where most languages do parsing via an external file format and a tool to translate that file format into native code (a la lex and yacc), Scala’s parser library makes extensive use of infix and postfix operators to provide a “traditional” syntax for describing a parser, but manages it entirely within the Scala language.  I’ll take a closer look at this in a future post.

March 29, 2009 in Software | Permalink | Comments (1) | TrackBack

March 25, 2009

Scala for C# programmers, part 5: implicits

It crushes my gentle spirit to call this article “implicits.”  How dreary.  The Scala wiki, describing but one use of implicits, uses the term “pimp my library,” which is so much better.  In the future, all language features will be named in this manner.  Extension methods, for example, will be called “pimp my interface.”  Generics will be called “pimp my type system.”  LINQ will be called “pimp my collections.”  The exceptions will be C++ templates and Lisp macros, which will be referred to using drugs terminology instead.

Terminological difficulties aside, Scala implicits offer some features which will be familiar to the C# programmer, and some which go way beyond pimping and into the aforesaid drugs metaphor.

Let’s start, however, with the pimping.

Extending Types in C# and Scala

Scala, like C#, is statically typed: a class’ methods are compiled into the class definition and are not open for renegotiation.  You cannot, as you might in Ruby or Python, just go ahead and declare additional methods on an existing class.

This is of course very inconvenient.  You end up declaring a load of XxxHelper or XxxUtils classes full of static methods, and having to write verbose calling code such as if (EnumerableUtils.IsEmpty(sequence)) rather than the rather more readable if (sequence.IsEmpty()).

C# 3 addressed this problem by introducing extension methods.  Extension methods are static methods in a XxxHelper or XxxUtils kind of class, except you’re allowed to write them using member syntax.  By defining IsEmpty as an extension method on IEnumerable, you can write if (sequence.IsEmpty()) after all.

Scala disapproves of static classes and global methods, so it plumps for an alternative approach.  You’ll still write a XxxHelper or XxxUtils kind of class, but instead of taking the Xxx to be Helped or Utilised as a method parameter, your class will wrap the Xxx.  Let’s see this in action as we try to add a method to the Double type:

class DoubleExtensions(d : Double) {
  def toThe(exp: Double): Double = System.Math.Pow(d, exp)

Notice that toThe is an instance method, and that DoubleExtensions takes a Double as a constructor parameter.  This seems pretty grim, because we’d normally have to access the function like this:

val result = new DoubleExtensions(2.0).toThe(7.0)

Hardly readable.  To make it look nice, Scala requires us to define an implicit conversion from Double to DoubleExtensions:

object Implicits {
  implicit def DoubleExtensions(d : Double) = new DoubleExtensions(d)

and to bring that implicit conversion into scope:

import Implicits._

Now we can write this:

val twoToTheSeven = 2.0.toThe(7.0)

and all will be well.  The Double type has apparently been successfully pimped with the toThe method.

This is, of course, just as much an illusion as the C# equivalent.  C# extension methods don’t add methods to a type, and nor do Scala implicit conversions.  What has happened here is that the Scala compiler has looked around for implicit methods that are applicable to the type of 2.0 (namely Double), and return a type that has a toThe method.  Our Implicits.DoubleExtensions method fits the bill, so the Scala compiler silently inserts a call to that method.  At runtime, therefore, Scala calls Implicits.DoubleExtensions(2.0) and calls the toThe of the resulting DoubleExtensions.

If setting this up seems a bit verbose, well, maybe.  C# extension methods are designed to be easily – one might even say implicitly – brought into scope.  That’s very important for operators like the LINQ standard query operators, but it can result in unwanted extension methods being dragged into scope and causing havoc.  Scala requires the caller to be a bit more explicit about implicits, which results in a slightly higher setup cost but gives the caller finer control over which implicit methods are considered.

But as it happens you can avoid the need for separate definitions of Implicits and DoubleExtensions, and get back to a more concise representation by using an anonymous class.  (As you’d expect, Scala anonymous classes are fully capable, like Java ones, rather than the neutered C# version.)  Here’s how it looks:

object Implicits {
  implicit def doubleToThe(d1 : Double) = new {
    def toThe(d2 : Double) : Double = Math.Pow(d1, d2)

(Unfortunately, the Scala .NET compiler currently seems to have some problems calling extension methods defined using anonymous classes, so this example only works with the JVM compiler, but you get the idea.)

Well, big deal.  Scala can pimp existing types with new methods just like C#, but using a different syntax.  In related news, Lisp uses a different kind of bracket: film at eleven.  Why should we be interested in Scala implicits if they’re just another take on extension methods?

Implicit Parameters

What we saw above was an implicit method – a method which, like a C# implicit conversion operator, the compiler is allowed to insert a call to without the programmer writing that call.  Scala also has the idea of implicit parameters – that is, parameters which the compiler is allowed to insert a value for without the programmer specifying that value.

That’s just optional parameters with default values, right?  Like C++ and Visual Basic have had since “visual” meant ASCII art on a teletype, and like C# is about to get?  Well, no.

C++, Visual Basic and C# optional parameters have fixed defaults specified by the called function.  For example, if you have a method like this:

public void Fie(int a, int b = 123) { … }

and you call Fie(456), it’s always going to be equivalent to calling Fie(456, 123).

A Scala implicit parameter, on the other hand, gets its value from the calling context.  That allows programmer calling the method to control the implicit parameter value, creating an extensibility point that optional parameters don’t provide.

This probably all sounds a bit weird, so let’s look at an example.  Consider the following Concatenate method:

public T Concatenate<T>(IEnumerable<T> sequence, T seed, Func<T, T, T> concatenator);

We pass this guy a sequence, a start value and a function that combines two values into one, and it returns the result of calling that function across the sequence.  For example, you could pass a sequence of strings, a start value of String.Empty, and (s1, s2) => s1 + s2, and it would return you all the strings concatenated together:

IEnumerable<string> sequence = new string[] { “mog”, “bites”, “man” };
string result = Concatenate(sequence, String.Empty, (s1, s2) => s1 + s2);
// result is “mogbitesman”

But this is a unpleasantly verbose.  We’re having to pass in String.Empty and (s1, s2) => s1 + s2 every time we want to concatenate a sequence of strings.  Not only is this tedious, it also creates the opportunity for error when the boss decides to “help” and passes the literal “String.Empty” as the seed value instead.  (“Oh, and I upgraded all the semi-colons to colons while I was in there.  No, don’t thank me!”)  We’d like to just tell the Concatenate function, “Look, this is how you concatenate strings,” once and for all.

Let’s start out by redefining the Concatenate method in Scala.  I’m going to factor out the seed and the concatenator method into a trait because we’ll typically be defining them together.

trait Concatenator[T] {
  def startValue : T
  def concat(x : T, y : T) : T

object implicitParameters {
  def concatenate[T](xs : List[T])(c : Concatenator[T]) : T =
    if (xs.isEmpty) c.startValue
    else c.concat(xs.head, concatenate(xs.tail)(c))

We can call this as follows:

object stringConcatenator extends Concatenator[String] {
  def startValue : String = ""
  def concat(x : String, y : String) = x.concat(y)

object implicitParameters {
  def main(args: Array[String]) = {
    val result = concatenate(List("mog", "bites", "man"))(stringConcatenator)

So far, this looks like the C# version except for the factoring out of the Concatenator trait.  We’re still having to pass in the stringConcatenator at the point of the call.  Let’s fix that:

def concatenate[T](xs : List[T])(implicit c : Concatenator[T]) : T =
  if (xs.isEmpty) c.startValue
  else c.concat(xs.head, concatenate(xs.tail))

We’ve changed two things here.  First, we’ve declared c to be an implicit parameter, meaning the caller can leave it out.  Second, we’ve, er, left it out ourselves, in the recursive call to concatenate(xs.tail).

Well, okay, it’s nice that concatenate now doesn’t have to pass the Concatenator explicitly to the recursive call, but we’re still having to pass in the stringConcatenator object to get things started.  If only there were some way to make the stringConcatenator object itself implicit…

object Implicits {
  implicit object stringConcatenator extends Concatenator[String] {
    def startValue : String = ""
    def concat(x : String, y : String) = x.concat(y)

Again, we’ve done two things here.  First, we’ve declared the stringConcatenator object implicit.  Consequently, we’ve had to move it out of the top level, because Scala doesn’t allow implicits at the top level (because they’d pollute the global namespace, being in scope even without an explicit import statement).

Now we can call concatenate like this:

import Implicits._

object implicitParameters {
  def main(args: Array[String]) = {
    val result = concatenate(List("mog", "bites", "man"))

And we’ll still get “mogbitesman” as the output.

Let’s review what’s going on here.  The implicit parameter of concatenate has been set to our stringConcatenator, a default value that the concatenate method knew nothing about when it was compiled.  This is somewhere north of what classical optional parameters are capable of, and we’re not finished yet.  Let’s build a listConcatenator.

object Implicits {
  class ListConcatenator[T] extends Concatenator[List[T]] {
    def startValue : List[T] = Nil
    def concat(x : List[T], y : List[T]) = x ::: y
  implicit object stringListConcatenator extends ListConcatenator[String] { }

This is a bit vexing.  List in Scala is a generic type, and has a generic concatenation operator called :::.  But we can’t create a generic object, because an object is an instance.  And implicit parameters have to be objects.  So the best we can do is build a generic ListConcatenator class, and then create trivial implicit objects for each generic parameter type we might need.  This seems a bit silly so I am probably missing something here, but this is the best I’ve managed so far.

However, accepting that I am dumb, let’s not worry about the ugly implementation, and see how this is used at the calling end:

val result = concatenate(List(
  List("mog", "bites", "man"),
  List("on", "beard")

This displays List(mog, bites, man, on, beard); that is, it concatenates the two List[String]s into one.  Once again, we have not had to pass stringListConcatenator explicitly: the Scala compiler has gone and found it for us.  We can use the exact same calling code to concatenate lists and strings.

Why Should I Care?

Isn’t this pointless?  At the call site, I have access to stringConcatenator and listStringConcatenator.  I can easily pass them in rather than relying on spooky compiler magic to do it for me.  Aren’t implicit parameters just job security for compiler writers?

Yes, implicit parameters are technically unnecessary.  But if we’re going to play that game, C# is technically unnecessary.  You could write all that code in IL.  Extension methods are unnecessary because you could write the static method out longhand.  Optional parameters are unnecessary because you could read the documentation and pass them in explicitly.  Post-It notes are unnecessary because you could fire up Outlook and create a Note instead.

Implicit parameters are about convenience and expressiveness.  Implicit parameters give you a way of describing how a function should handle different situations, without needing to bake those situations into the function logic or to specify them every damn time you call the function.  You don’t want to have to tell the concatenate function whether to use the List or String concatenator every time you call it: the compiler knows what you’re concatenating; specifying how to concatenate it just gives you a chance to get it wrong!

Consequently, implicit parameters – like implicit conversions – contribute to Scala’s ability to support internal DSLs.  By setting up appropriate implicits, you can write code that reads much more naturally than if you had to pepper it with function objects or callbacks.


Scala’s implicit keyword goes somewhat beyond C#’s equivalent.  As in C#, it is used for implicit conversions; unlike C#, this is the idiomatic way to add operations to an existing type, removing the need for the separate extension method syntax.  Implicit parameters have no equivalent in C#.  They are like being able to add default values to a method: just as a C# using statement bring implicit methods into scope, a Scala import statement can bring default values into scope.  If implicit conversions are a way of extending classes, then implicit parameters are a way of extending methods, creating simple, reliable shorthands for complex generic methods, and making up another piece of the Scala DSL jigsaw.

And the big terminological question.  Where do Scala implicits sit on the pimp–drugs scale of metaphors?  Pimping, for now.  But maybe dealing a little on the side, and keeping an eye on the main chance.

March 25, 2009 in Software | Permalink | Comments (2) | TrackBack

March 01, 2009

Small businesses and "fire at will"

Idiot/Savant at No Right Turn observes that the Employment Relations Amendment Act – the “fire at will” law that allows small businesses to sack employees within the first 90 days with no process and no reason – comes into force today, and advises, “If you have any sort of choice at all, I suggest not working for a small business.”

This is an obnoxious law, but this is terrible advice on how to respond to it.

Yes, there will be small businesses which abuse their employees by using this law to sack people because they fall sick or complain about discrimination or decline to work overtime, or just for no reason at all.

But many, many small businesses will want nothing to do with this kind of behaviour.  My own employer, Mindscape, is a small business – currently the two directors and me – and I find it inconceivable that we would use this law in an abusive way.  Wellington is a small town: word of abuse would spread almost instantly around our potential employee pool.  We want high-value people, and high-value people won’t waste their time on an employer if that employer has a reputation for screwing people around.  It’s bad, bad advice to suggest that people should choose an EDS or a Datacom over a Mindscape just because we’re covered by the “fire at will” law and the big companies aren’t.

Obviously, different businesses and different locations will be more or less affected by such considerations.  A small manufacturing shop in Auckland employing semi-skilled labour could get away with abuse for a far longer time than a software development shop in Wellington.  (Though it would be interesting to see if an “ERAA Watch”-type Web site a la NoCowboys could gain any traction in warning people about abusive employers.)  Nevertheless, small businesses tend on average to need higher-value employees, and these people often do have a choice of where to work, and will therefore choose an honest employer over one with a track record of abuse.  The mantra of “you don’t shit where you eat” remains an important one for businesses of all sizes.

Small businesses are a lot of fun to work for, much more than big businesses, and with plenty more opportunities for their employees.  I’d hate to see people put off working for great small businesses like Mindscape just because this obnoxious law gives free rein to the bad ones.  Better advice would be, “If you have any sort of choice at all, I suggest not working for an arsehole.”  And that applies no matter what the size of the business.

[Note: I usually don’t bother with this disclaimer because it’s bleedin’ obvious, but since I used the word “we” when talking about Mindscape I should probably make it clear that I am speaking only for myself, not for my employer.]

March 1, 2009 in Current Affairs | Permalink | Comments (3) | TrackBack