« Scala for C# programmers, part 2: singletons | Main | Wellington .NET user group: C# 4 and CLR 4.0 »

January 10, 2009

Scala for C# programmers, part 3: pass by name

"You're only on part 3 and you're already reduced to writing about calling conventions?  You suck!  Do another post about chimney sweeps being hunted by jars of marmalade!"

Silence, cur.  Pass by name is not as other calling conventions are.  Pass by name, especially in conjunction with some other rather theoretical-sounding Scala features, is your gateway to the wonderful world of language extensibility.

What is Passing By Name?

First, let's talk about what we mean by calling convention.  A calling convention describes how stuff gets passed to a method by its caller.  In the good old days, this used to mean exciting things like which arguments got passed in registers and who was responsible for resetting the stack pointer.  Sadly, the days of being able to refer to "naked fun calls" and make the Beavis and Butthead noise are consigned to history: in modern managed environments, the runtime takes care of all this guff and the main distinction is pass data by value or by reference.  (The situation on the CLR is slightly complicated by the need to differentiate passing values by value, values by reference, references by value and references by reference, but I'm not going to go into that because (a) it's irrelevant to the subject at hand and (b) that's Jon Skeet's turf and I don't want him to shank me.  Again.)

In pass by value, the called method gets a copy of whatever the caller passed in.  Arguments passed by value therefore work like local variables that are initialised before the method runs: when you do anything to them, you're doing it to your own copy.

In pass by reference, the called method gets a reference to the caller's value.  When you do anything to a pass-by-reference argument, you're doing it to the caller's data. 

In pass by name, the called method gets... well, it's a bit messy to explain what the called method gets.  But when the called method does anything to the argument, the argument gets evaluated and the "anything" is done to that.  Crucially, evaluation happens every time the argument gets mentioned, and only when the argument gets mentioned.

Not Just Another Calling Convention

Why does this matter?  It matters because there are functions you can't implement using pass by value or pass by reference, but you can implement using pass by name.

Suppose, for example, that C# didn't have the while keyword.  You'd probably want to write a method that did the job instead:

public static void While(bool condition, Action body)
{
  if (condition)
  {
    body();
    While(condition, body);
  }

}

What happens when we call this?

long x = 0;
While(x < 10, () => x = x + 1);

C# evaluates the arguments to While and invokes the While method with the arguments true and () => x = x + 1.  After watching the CPU sit on 100% for a while you might check on the value of x and find it's somewhere north of a trillion.  Why?  Because the condition argument was passed by value, so whenever the While method tests the value of condition, it's always true.  The While method doesn't know that condition originally came from the expression x < 10; all While knows is that condition is true.

For the While method to work, we need it to re-evaluate x < 10 every time it hits the condition argument.  While needs not the value of the argument at the call site, nor a reference to the argument at the call site, but the actual expression that the caller wants it to use to generate a value.

Same goes for short-circuit evaluation.  If you want short-circuit evaluation in C#, your only hope if to get on the blower to Anders Hejlsberg and persuade him to bake it into the language:

bool result = (a > 0 && Math.Sqrt(a) < 10);
double result = (a < 0 ? Math.Sqrt(-a) : Math.Sqrt(a));

You can't write a function like && or ?: yourself, because C# will always try to evaluate all the arguments before calling your function.  Consider a VB exile who wants to reproduce his favourite keywords in C#:

bool AndAlso(bool condition1, bool condition2)
{
  return condition1 && condition2;
}

T IIf<T>(bool condition, T ifTrue, T ifFalse)
{
  if (condition)
    return ifTrue;
  else
    return ifFalse;
}

But when C# hits one of these:

bool result = AndAlso(a > 0, Math.Sqrt(a) < 10);
double result = IIf(a < 0, Math.Sqrt(-a), Math.Sqrt(a));

it would try to evaluate all the arguments at the call site, and pass the results of those evaluations to AndAlso or IIf.  There's no short-circuiting.  So the AndAlso call would crash if a were negative, and the IIf call if a were anything other than 0.  Again, what you want is for the condition1, condition2, ifTrue and ifFalse arguments to be evaluated by the callee if it needs them, not for the caller to evaluate them before making the call.

And that's what pass by name does.  A parameter passed by name is not evaluated when it is passed to a method.  It is evaluated -- and re-evaluated -- when the called method evaluates the parameter; specifically when the called method requests the value of the parameter by mentioning its name.  This might sound weird and academic, but it's the key to being able to define your own control constructs.

Using Pass By Name in Scala

Let's see the custom while implementation again, this time with Scala "pass by name" parameters:

def mywhile(condition: => Boolean)(body: => Unit): Unit =
  if (condition) {
    body
    mywhile(condition)(body)
  }

We can call this as follows:

var i = 0
mywhile (i < 10) {
  println(i)
  i += 1
}

Unlike the C# attempt, this prints out the numbers from 0 to 9 and then terminates as you'd wish.

Pass by name also works for short-circuiting:

import System.Math._

def andAlso(condition1: => Boolean, condition2: => Boolean): Boolean =
  condition1 && condition2

val d = -1.234
val result = andAlso(d > 0, Sqrt(d) < 10)

The andAlso call returns false rather than crashing, because Sqrt(d) < 10 never gets evaluated.

There Are No Miracles in Our Profession, Trurl!

What's going on here?  What's the weird colon-and-pointy-sticks syntax?  What is actually getting passed to mywhile and andAlso to make this work?

The answer is a bit surprising.  Nothing is going on here.  This is the normal Scala function parameter syntax.  There is no "pass by name" in Scala.

Here's a bog-standard pass by value Scala function declaration:

def myFunc1(i: Int) : Unit = ...

Takes an integer, returns void: easy enough.  Here's another:

def myFunc2(f: Int => Boolean) : Unit = ...

Even if you've not seen this kind of expression before, it's probably not too hard to guess what this means.  This function takes a function from Int to Boolean as its argument.  In C# terms, void MyFunc2(Func<int, bool> f).  We could call this as follows:

myFunc2 { (i: Int) => i > 0 }

So now you can guess what this means:

def myFunc3(f: => Boolean) : Unit = ...

Well, if myFunc2 took an Int-to-Boolean function, myFunc3 must be taking a blank-to-Boolean function -- a function that takes no arguments and returns a Boolean.  In short, a conditional expression.  So we can call myFunc3 as follows:

val j = 123
myFunc3 { j > 0 }

The squirly brackets are what we'd expect from an anonymous function, and because the function has no arguments Scala doesn't make us write { () => j > 0 }, even though that's what it means really.  The anonymous function has no arguments because j is a captured local variable, not an argument to the function.  But there's more.  Scala also lets us call myFunc3 like this:

val j = 123
myFunc3(j > 0)

This is normal function call syntax, but the Scala compiler realises that myFunc3 expects a nullary function (a function with no arguments) rather than a Boolean, and therefore treats myFunc3(j > 0) as shorthand for myFunc3(() => j > 0).  This is the same kind of logic that the C# compiler uses when it decides whether to compile a lambda expression to a delegate or an expression tree.

You can probably figure out where it goes from here:

def myFunc4(f1: => Boolean)(f2: => Unit) : Unit = ...

This takes two functions: a conditional expression, and a function that takes no arguments and returns no value (in .NET terms, an Action).  Using our powers of anticipation, we can imagine how this might be called using some unholy combination of the two syntaxes we saw for calling myFunc3:

val j = 123;
myFunc4(j > 0) { println(j); j -= 1; }

We can mix and match the () and {} bracketing at whim, except that we have to use {} bracketing if we want to batch up multiple expressions.  For example, you could legally equally well write the following:

myFunc4 { j > 0 } { println(j); j -= 1; }
myFunc4 { println(j); j > 0 } (j -= 1)
myFunc4 { println(j); j > 0 } { j -= 1 }

And we'll bow to the inevitable by supplying a body for this function:

def myFunc5(f1: => Boolean)(f2: => Unit) : Unit =
  if (f1()) {
    f2()
    myFunc5(f1)(f2)
  }

Written like this, it's clear that f1 is getting evaluated each time we execute the if statement, but is getting passed (as a function) when myFunc5 recurses.  But Scala allows us to leave the parentheses off function calls with no arguments, so we can write the above as:

def myFunc5(f1: => Boolean)(f2: => Unit) : Unit =
  if (f1) {
    f2
    myFunc5(f1)(f2)
  }

Again, type inference allows Scala to distinguish the evaluation of f1 in the if statement from the passing of f1 in the myFunc5 recursion.

And with a bit of renaming, that's mywhile.  There's no separate pass by name convention: just the usual closure behaviour of capturing local variables in an anonymous method or lambda, a bit of syntactic sugar for nullary functions (functions with no arguments), just like C#'s syntactic sugar for property getters, and the Scala compiler's ability to recognise when a closure is required instead of a value.

In fact, armed with this understanding of the Scala "syntax," we can easily map it back to C#:

void While(Func<bool> condition, Action body)
{
  if (condition())
  {
    body();
    While(condition, body);
  }
}

int i = 0;
While(() => i < 10, () =>
{
  Console.WriteLine(i);
  ++i;
});

The implementation of the While method in C# is, to my eyes, a bit clearer than the Scala version.  However, the syntax for calling the While method in C# is clearly way more complicated and less natural than the syntax for calling mywhile in Scala.  Calling mywhile in Scala was like using a native language construct.  Calling While in C# required a great deal of clutter at the call site to prevent C# from trying to treat i < 10 as a once-and-for-all value, and to express the body at all.

So that's so-called "pass by name" demystified: the Scala Web site, with crushing mundanity, demotes it to "automatic type-dependent closure construction," which is indeed exactly how it works.  As we've seen, however, this technical-sounding feature is actually essential to creating nice syntax for your own control constructs.  We'll shortly see how this works together with other Scala features to give you even more flexibility in defining your construct's syntax.

January 10, 2009 in Software | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c5c9b53ef010536c2ec6c970c

Listed below are links to weblogs that reference Scala for C# programmers, part 3: pass by name:

Comments

The part about C# not being able to short circuit isn't true.

http://msdn.microsoft.com/en-us/library/2a723cdk.aspx

Posted by: Lomohai at Jul 28, 2009 3:22:26 AM

I'm aware that C#'s built-in boolean operators short-circuit. The context of the comment was creating *custom* short-circuiting methods/operators in C#, which *isn't* possible.

Posted by: Ivan at Jul 28, 2009 7:00:44 AM

Right, thanks for the clarification.

Posted by: Lomohai at Jul 29, 2009 1:32:53 AM

Interesting that this is in Scala, I was wishing it was in C# when I started using lambdas a lot. But it's important to recognise that it is only syntactic sugar - so claims like "there are functions you can't implement..." are not really true. In C#, the caller has to put '() =>' in front of their expression argument, and the receiver has to add '()' after the parameter name to evaluate it.

They _could_ be made optional in the same way in C#, but consider the 'out' keyword. The caller has to put 'out' in front of the argument. This is for a reason: so that when you read the code, it is clear that the variable may be modified by the method.

Requiring the '() =>' probably serves the same purpose, with respect to side-effects. In languages with side-effects, it is probably a good thing to be able to clearly see what you are handing over to another module: is it a value, or is something to be executed later to produce a value (and possibly side effects too?)

Posted by: Daniel Earwicker at Sep 26, 2009 12:17:35 AM

Re "there are functions you can't implement": true, you can implement them with the assistance of the caller, and if you accept the calamitous effect of the lambda wrapping on readability at the call site. But you can't implement them in the way the C# language does. Compare the example of While as a custom function (in the last paragraph but two) to the while language keyword.

I agree that automatic closures ("pass by name") could be misused, but that's down to being clear about under what circumstances any given argument will be evaluated. I don't think anybody would be surprised by the custom While function even if we removed the ()=> noise from it. Sure, explicitness is a good thing, but language design has to balance this against clutter and noise (and flexibility/extensibility). One of the interesting things about comparing C#/Java and Scala is seeing the effects of different language design philosophies -- C# definitely favours explicitness, exactitude and obviousness, whereas Scala tends to prefer to enable more concise and expressive notation even if this may require more learning and analysis to understand program flow. The C# philosophy isn't wrong: it's appropriate for C#'s target market. Unfortunately, this makes it hard to create expressive notations even when such notations would repay the additional analysis / learning cost for readers and maintainers of the code. Really must get around to writing about this...

Posted by: Ivan at Sep 26, 2009 8:41:49 AM

All very true. My experience with C++ was that with a mix of templates, operator overloading and so on, I could make a library that came very close to the syntax I wanted, but which none of my colleagues can maintain to this day :)

I must experiment more with Scala to see if it as "risky" as I imagine!

Posted by: Daniel Earwicker at Oct 2, 2009 10:31:48 AM

Nice article, thanks for sharing..

Posted by: rana at Dec 27, 2009 5:35:32 PM