Scala - where are are - where we plan to go

1. Scala- where are are- where we plan to go Martin Odersky

2. Welcome to ScalaDays! • 155 attendees, coming from all corners of the planet. • We are packed. Sadly, many more people could not be admitted for lack of space. • This shows the drive and energy of this community! • The Scala team is looking forward to talk with you and learn from you during the next two days. • We are particularly glad to have you here at EPFL, the place where Scala was born.

3. History (incomplete) 2001 Scala design begins, as a more practical successor to Funnel. 2003 First experimental release, first course taught with it. 2004 Scalable Component Abstractions paper. 2005 Scala 2.0, written in Scala. 2006 Industrial adoption starts. 2007 First Lift release. 2008 First Scala LiftOff, Twitter adopts Scala. 2009 Big increase in adoption, IDEs mature. 2010 First ScalaDays.

4. The main idea behind Scala

5. Scala is a Unifier Agile, with lightweight syntax Object-Oriented Scala Functional Safe and performant, with strong static typing

6. The Philosophy behind Scala Put productivity and creativity back in the hands of developers. “Joy is underrated as a metric for a language’s potential success in a development organization.” a3lx@twitter - address professional developers - trust them & value their expertise - (don’t tell them how they should do their job)

7. Some users in industry

8. Open Source Ecosystem Many great projects, including: lift (David P. & friends) akka (Jonas B.) kestrel, cachet, queroulous (Twitter) migrations (Sony) sbt (Mark H.) ScalaTest (Bill V. & friends) specs, ScalaCheck, ScalaModules, ScalaQL, .... ( the list goes on, many more tools & projects presented at this conference )

9. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Now: 2.8 First release candidate Scala 2.8 RC1 is out! This is primarily a consolidating release • Some new language features • More complete libraries • Redesigns of a few features that did not work out well. • Big improvements in tooling. • Big improvements in stability. • Laying the groundwork for the future.

10. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Named and Default Parameters defnewTable(size: Int = 100, load: Float = 0.5f) = newHashMap[String, String]{ overridevalinitialSize = size overridevalloadFactor = (load* 1000).toInt } } newTable(50, 0.75f) newTable() newTable(load = 0.5f) newTable(size = 20, load = 0.5f) • Defaults can be arbitrary expressions, not just constants. • Default evaluation is dynamic.

11. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Copy Operation Problem: Case classes are great for pattern matching,but how do you transform data? Often need to update (functionally) some specific field(s)of a date structure defined by case classes. Solution: Selective copy method. caseclass Tree[+T](elem: T, left: Tree[T], right: Tree[T]) def incElem(t: Tree[Int]) = t copy (elem = t.elem + 1) Q: How to define “copy”?

12. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Defining “copy” Problem: A class with n parameters allows 2n possible combinations of parameters to copy. Default parameters to the rescue. case class Tree[+T](elem: T, left: Tree[T], right: Tree[T]) { // generated automatically: defcopy[U](elem: U = this.elem, left: Tree[U] = this.left, right: Tree[U] = this.right) = Branch(elem, left, right) }

13. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Nested Annotations • Scala 2.7 largely followed Java annotation syntax, but left out some of the more convoluted constructions.  Nested annotations were not possible. • Scala 2.8 throws out (deprecates) all special annotation syntax. • Instead uses class constructors, which can have named parameters. • This greatly simplifies the syntax, and allows all annotations to be expressed, including nested ones. @JoinTable( joinColumns = Array(new JoinColumn(name = "a name“)) var test: String = _

14. Scala 2.8 named and default parameters copy operation nested annotations @specialized new collection API vectors hash tries new treatment of arrays and strings package objects safer package nesting manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL @specialized Problem: Generics cause a performance hit because they require a uniform representation: everything needs to be boxed. Example: trait Function1[-T, +U] { def apply(x: T): U } def app(x: Int, fn: Int => Int) = fn(x) app(7, x => x * x) Translatedby scalac 2.7 to: trait Function1 { def apply(x: Object): Object } class anonfun extends Function1 { def apply(x: Object): Object = Int.box(apply(Int.unbox(x))) def apply(x: Int): Int = x * x} def app(x: Int, fn: Function1) = Int.unbox(fn(Int.box(x))) app(7, new anonfun)

15. Scala 2.8 named and default parameters copy operation nested annotations @specialized new collection API vectors hash tries new treatment of arrays and strings package objects safer package nesting manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL If Function1 is @specialized ... trait Function1[@specialized -T, @specialized +U] { def apply(x: T): U } def app(x: Int, fn: Int => Int) = fn(x)app(7, x => x * x) ... scalac 2.8 generates instead: trait Function1 { def apply(x: Object): Object def apply$$II(x: Int): Int = Int.unbox(apply(Int.box(x))) // specializations for other primitive types}class anonfun extends (Int => Int) { def apply(x: Object): Object = Int.box(applyIntInt(Int.unbox(x))) override def apply$$II(x: Int): Int = x * x} def app(x: Int, fn: Function1) = fn.apply$II(x) app(7, anonfun) No Boxing!

16. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Scala Collections Collection Properties: object-oriented generic:List[T], Map[K, V] optionally persistent, e.g. collection.immutable.Seq higher-order, with methods such as foreach, map, filter. Uniform return type principle: Operations return collections of the same type (constructor) as their left operand. scala> val ys = List(1, 2, 3) ys: List[Int] = List(1, 2, 3) scala> val xs: Seq[Int] = ys xs: Seq[Int] = List(1, 2, 3) scala> xs map (_ + 1) res0: Seq[Int] = List(2, 3, 4) scala> ys map (_ + 1) res1: List[Int] = List(2, 3, 4)

17. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Old Collection Structure traitIterable[A] { def filter(p: A => Boolean): Iterable[A] = ... def partition(p: A => Boolean) = (filter(p(_)), filter(!p(_))) def map[B](f: A => B): Iterable[B] = ... } traitSeq[A] extends Iterable[A] { def filter(p: A => Boolean): Seq[A] = ... overridedef partition(p: A => Boolean) = (filter(p(_)), filter(!p(_))) def map[B](f: A => B): Seq[B] = ... } Types force duplication!

18. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL New Collection API • Avoids type & code duplication through • uniform framework of traversers and builders. • supported by: implicits, higher-kinded types • Uniform package organization reflects mutability: • scala.collection • scala.collection.mutable • scala.collection.immutable • Richer and more regular set of collection operations. • New collection implementations: vectors and hashtries. • More info: “Fighting bit rot with types”, FSTTCS09

19. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Vectors and Hash Tries • Trees with branch factor of 32. • Persistent data structures with very efficient sequential and random access. • Invented by Phil Bagwell, then adopted in Clojure. • New: Persistent prepend/append/update in constant amortized time. • Next: Fast splits and joins for parallel transformations.

20. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Package Objects Motivation: • The new collection design reshuffled a lot of classes. • For instance: scala.List  scala.collection.immutable.List • How can we keep the old aliases around? • How can we migrate old to new? • Easy: put type List = scala.collection.immutable.List val List = scala.collection.immutable.List into the scala package. • But how to we put a type alias and a val definition into a package?

21. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Package Objects Problem: Everything has to live inside an object or class.=> No top-level value definitions, type aliases, implicits. Package objects make packages more like objects. They also let you define your own ``Predefs’’. package object scala { type List[+T] = scala.collection.immutable.List val List = scala.collection.immutable.List ... }

22. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Safer Package Nesting • Java knows only absolute packages • But Scala’s packages nest • This can lead to an impedance mismatch. • For instance: package net.liftweb java.lang.System.print(“hello”) • This used to fail if net contains a subpackage java. • New rules: Intermediate packages only visible in multiple package clauses. • To access net.java as java, you’d need to write package net package liftweb ...

23. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL New Treatment of Arrays • Arrays were “magical” up to 2.8 • sometimes primitive • sometimes wrapped • automatic boxing/unboxing conversions based on static type. • This could be confusing and slow. • 2.8 removes the magic. • Arrays are always Java arrays • Two implicit conversions make sure that • arrays are compatible with sequences. • arrays support all sequence operations • Strings are treated similarly to arrays.

24. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Manifests • Question: How to create an Array[T] where T is a type parameter? • No magic allowed! • Solution: supply the run-time type as a type descriptor called a Manifest or ClassManifest: def evens[T: ClassManifest](xs: Vector[T]): Array[T] = { val arr = new Array[T]((xs.length + 1) / 2) for (i <- 0 until xs.length by 2) arr(i / 2) = xs(i) arr } The first line is equivalent to: def evens[T](xs: Vector[T]) (implicit ev: ClassManifest[T]): Array[T]

25. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Better ImplicitsBetter Type Inference • Type inference for implicits has been refined. • Previously: Infer type first, search implicit for inferred type afterwards. • Now: Implicit search can drive type inference. • This makes implicits more powerful. • Other advance: Type constructors of higher-kinded types can now be inferred.

26. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference delimited continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Delimited Continuations • Continuations capture the return stack as a data structure. • Delimited continuations capture only up to some enclosing scope. • Lost of advanced control structures can be built on them, e.g: • coroutines • event-based I/O • reactive programming • lightweight actors • Scala 2.8 supports delimited continuations in a compiler plugin.

27. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Better IDEs • The Scala plugins for Eclipse, IntelliJ, Netbeans all have matured a lot. • The Eclipse and Netbeans plugins are now based on completely redesigned compiler interfaces for command completion, semantic highlighting and incremental building. • If you have discounted IDEs before for Scala, maybe now is the time to try again!

28. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Docs for Advanced Types Problem: Advanced statically typed APIs pose challenges for presentation and documentation.

29. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL Simpler Type Views • How to explain def map[B, To](f: A => B) (implicit cbf: CanBuildFrom[Coll, B, To]): To to a beginner? • Key observation: We can approximate the type of map. • For everyone but the most expert user def map[B](f: A => B): Traversable[B] // in class Traversabledef map[B](f: A => B): Seq[B] // in class Seq, etc is detailed enough. • These types are correct, they are just not as general as the type that’s actually implemented. • These are documented in the new ScalaDoc as use cases.

30. Scala 2.8 named and default parameters copy operation nested annotations package objects safer package nesting @specialized new collection API vectors hash tries new treatment of arrays and strings manifests better implicits better type inference continuations packrat parsing faster actors improved Swing support better IDEs interactive compiler API new scaladoc better REPL New ScalaDoc

31. Why 2.8, not 3.0? • Scala 2.8 is numerically a point release, but feels like a major release. • So why did we not name it Scala 3.0? • 2.8 has been 18 months in the making, and acquired more changes on the way. • Originally, we just wanted to rewrite collections. • That took longer than expected. • Meanwhile, a lot more additions and changes got included. • But 2.8 was already announced, and referred to in books and other documentation. • In light of this, we judged it too confusing to change the version number in mid-flight.

32. And now... ...what’s next?

33. Next: Scala on .NET • Work started again to have a native Scala compiler on .NET with Visual Studio integration. • Funded my Microsoft. • Stay tuned...

34. Next: Reliability and Contracts • Lost of things happening around contracts and verification. • Peter Müller: Scala verifier • Viktor Kuncak: Scala code synthesis from constraints • Our group: effects • Common project: ProgLab.NET

35. Next: Concurrency • Actor encapsulation • How to ensure that actors do not share memory? • Solved by capability-based uniqueness type system. • Implemented in compiler plugin. • Paper in ECOOP 2010. • Software Transactional Memory • CCSTM work in this conference • Open question: What’s a good combination of actors and transactions?

36. Next: Parallelism • PPP Challenge: How to make use of multi-cores, GPUs, clusters, clouds, to achieve better performance for normal applications? • First step in this direction: Parallel collections. • will be integrated in the new collection API. • based on persistent data structures with fast splits and joins. • Longer-term vision: Embedded parallel DSLs. • See keynote by Kunle Olokutun tomorrow.

37. Next: The Green House Great work on Scala is springing up everywhere, no longer focussed just at EPFL. Q: How can we integrate contributions? Tension: Want to get lots of exciting new stuff contributed, yet maintain a stable and reliable distribution. Planned Solution: three-tiered: Incubator Hosts collaborative Scala projects Green house A “cutting edge” version of the distribution with experimental features. Trunk The stable distribution.

38. Thanks • To the Scala team at EPFL: Lukas Rytz, Hubert Plociniczak, Iulian Dragos, Miguel Garcia, Gilles Dubochet, Philipp Haller, Aleksandar Prokopec, Antonio Cunei, Tiark Rompf, Donna Malayeri, Phil Bagwell, Adriaan Moors, Ingo Maier. • To our external committers and frequent contributors: Paul Phillips, Miles Sabin, Ilya Sergey, Caoyuan Deng, James Matlik, Frank Teubler, Kevin Wright, Manohar Jonnalagedda, Pedro Furlanetto, Johannes Rudolph, Jason Zaugg, Seth Tisue, Ismael Juma, Mark Harrah, Colin Howe, Mirko Stocker, Spiros Tzavellas, Matt Russell, David Taylor • To the community at large for your support, suggestions, encouragements.