Scala Macros is an indispensable programming tool for scala library programmers. It can solve some problems that can not be solved by ordinary programming or class level programming, because Scala Macros can modify programs directly. Scala Macros works by modifying a program to produce a new program according to the programmer's intention at the time of program compilation. The specific process is: when the compiler finds the Macro tag, it will pull the Macro's function implementation: an AST (Abstract Syntax Tree) structure to replace the Macro's position, and then proceed with the type verification process from the AST.
Let's start with a simple example to demonstrate and analyze the basic principles and usage of Def Macros.
1 object modules {
2 greeting("john")
3 }
4
5 object mmacros {
6 def greeting(person: String): Unit = macro greetingMacro
7 def greetingMacro(c: Context)(person: c.Expr[String]): c.Expr[Unit] = ...
8 }
These are the standard implementation patterns of Def Macros. The basic principle is as follows: when compiler encounters method call greeting("john") in compiling modules, it parses function symbols, and finds greeting is a macro in mmacros. Its concrete implementation is in greeting Macro function. At this time, the compiler runs greeting Macro function and calls an AST replacement expression greeting("john"). Note that the compiler passes in the parameter person in AST mode when calculating greetingMacro. Because greetingMacro functions need to be computed when compiling modules objects, greetingMacro functions and even the entire mmacros object must be compiled. This means that modules and mmacros must be in different source files, and also ensure that mmacros is compiled before compiling modules. We can see the relationship between them from the SBT settings file build.sbt:
1 name := "learn-macro"
2
3 version := "1.0.1"
4
5 val commonSettings = Seq(
6 scalaVersion := "2.11.8",
7 scalacOptions ++= Seq("-deprecation", "-feature"),
8 libraryDependencies ++= Seq(
9 "org.scala-lang" % "scala-reflect" % scalaVersion.value,
10 "org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.1",
11 "org.specs2" %% "specs2" % "2.3.12" % "test",
12 "org.scalatest" % "scalatest_2.11" % "2.2.1" % "test"
13 ),
14 addCompilerPlugin("org.scalamacros" % "paradise" % "2.1.0" cross CrossVersion.full)
15 )
16
17 lazy val root = (project in file(".")).aggregate(macros, demos)
18
19 lazy val macros = project.in(file("macros")).settings(commonSettings : _*)
20
21 lazy val demos = project.in(file("demos")).settings(commonSettings : _*).dependsOn(macros)
Note the last line: demos dependsOn(macros), because we put all macros definition files in the macros directory.
Let's look at the implementation of macro.
1 import scala.language.experimental.macros
2 import scala.reflect.macros.blackbox.Context
3 import java.util.Date
4 object LibraryMacros {
5 def greeting(person: String): Unit = macro greetingMacro
6
7 def greetingMacro(c: Context)(person: c.Expr[String]): c.Expr[Unit] = {
8 import c.universe._
9 println("compiling greeting ...")
10 val now = reify {new Date().toString}
11 reify {
12 println("Hello " + person.splice + ", the time is: " + new Date().toString)
13 }
14 }
15 }
These are the concrete declarations and implementations of macro greeting. The code is placed in MacrosLibrary.scala in the macros directory. First you must import macros and Context.
The macro call is in HelloMacro.scala in the demo directory:
1 object HelloMacro extends App {
2 import LibraryMacros._
3 greeting("john")
4 }
Note the output generated when compiling HelloMacro.scala:
Mac-Pro:learn-macro tiger-macpro$ sbt
[info] Loading global plugins from /Users/tiger-macpro/.sbt/0.13/plugins
[info] Loading project definition from /Users/tiger-macpro/Scala/IntelliJ/learn-macro/project
[info] Set current project to learn-macro (in build file:/Users/tiger-macpro/Scala/IntelliJ/learn-macro/)
> project demos
[info] Set current project to demos (in build file:/Users/tiger-macpro/Scala/IntelliJ/learn-macro/)
> compile
[info] Compiling 1 Scala source to /Users/tiger-macpro/Scala/IntelliJ/learn-macro/macros/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.8. Compiling...
[info] Compilation completed in 7.876 s
[info] Compiling 1 Scala source to /Users/tiger-macpro/Scala/IntelliJ/learn-macro/demos/target/scala-2.11/classes...
compiling greeting ...
[success] Total time: 10 s, completed 2016-11-9 9:28:24
>
From compiling greeting.... this prompt we can conclude that the greeting Macro function should be calculated in compiling the source code files in the demo directory. The results of the test run are as follows:
Hello john, the time is: Wed Nov 09 09:32:04 HKT 2016
Process finished with exit code 0
The operation greeting actually calls the macro implementation code in greetingMacro. The above example uses the most basic Scala Macro programming pattern. Notice in this example the parameter c: Context of the function greetingMacro and the call of reify, splice in the function internal code: Because Context is a dynamic function interface, each instance is different. For large macro implementations, other helper function s that also use Context may be invoked, which is prone to mismatch of Context instances. In addition, reify and splice can be said to be the most original AST operation functions. We use the latest models and methods in the following example:
1 def tell(person: String): Unit = macro MacrosImpls.tellMacro
2 class MacrosImpls(val c: Context) {
3 import c.universe._
4 def tellMacro(person: c.Tree): c.Tree = {
5 println("compiling tell ...")
6 val now = new Date().toString
7 q"""
8 println("Hello "+$person+", it is: "+$now)
9 """
10 }
11 }
In this example, we put the macro implementation function into a class with Context as its parameter. We can put all functions using Context in this class and share a unified Context instance. quasiquotes is the latest set of AST operation functions, which can control the generation of AST and expression restoration more conveniently and flexibly. The call to tell macro is the same:
1 object HelloMacro extends App {
2 import LibraryMacros._
3 greeting("john")
4 tell("mary")
5 }
The test operation produces the following results:
Hello john, the time is: Wed Nov 09 11:42:21 HKT 2016
Hello mary, it is: Wed Nov 09 11:42:20 HKT 2016
Process finished with exit code 0
The Macro implementation function of Def Macros can be a generic function that supports class parameters. In the following example, we demonstrate how to use Def Macros to implement the conversion of generic case class to Map type. Suppose we have a converter CaseClassMapConverter[C], then C type can be any case class, so this converter is generic, so macro implementation function must be generic. Generally speaking, we want to achieve the following functions: turn any case class into Map:
1 def ccToMap[C: CaseClassMapConverter](c: C): Map[String,Any] =
2 implicitly[CaseClassMapConverter[C]].toMap(c)
3
4 case class Person(name: String, age: Int)
5 case class Car(make: String, year: Int, manu: String)
6
7 val civic = Car("Civic",2016,"Honda")
8 println(ccToMap[Person](Person("john",18)))
9 println(ccToMap[Car](civic))
10
11 ...
12 Map(name -> john, age -> 18)
13 Map(make -> Civic, year -> 2016, manu -> Honda)
Reverse Map to case class:
1 def mapTocc[C: CaseClassMapConverter](m: Map[String,Any]) =
2 implicitly[CaseClassMapConverter[C]].fromMap(m)
3
4 val mapJohn = ccToMap[Person](Person("john",18))
5 val mapCivic = ccToMap[Car](civic)
6 println(mapTocc[Person](mapJohn))
7 println(mapTocc[Car](mapCivic))
8
9 ...
10 Person(john,18)
11 Car(Civic,2016,Honda)
Let's look at the Macro implementation function: macros/CaseClassConverter.scala
1 import scala.language.experimental.macros
2 import scala.reflect.macros.whitebox.Context
3
4 trait CaseClassMapConverter[C] {
5 def toMap(c: C): Map[String,Any]
6 def fromMap(m: Map[String,Any]): C
7 }
8 object CaseClassMapConverter {
9 implicit def Materializer[C]: CaseClassMapConverter[C] = macro converterMacro[C]
10 def converterMacro[C: c.WeakTypeTag](c: Context): c.Tree = {
11 import c.universe._
12
13 val tpe = weakTypeOf[C]
14 val fields = tpe.decls.collectFirst {
15 case m: MethodSymbol if m.isPrimaryConstructor => m
16 }.get.paramLists.head
17
18 val companion = tpe.typeSymbol.companion
19 val (toParams,fromParams) = fields.map { field =>
20 val name = field.name.toTermName
21 val decoded = name.decodedName.toString
22 val rtype = tpe.decl(name).typeSignature
23
24 (q"$decoded -> t.$name", q"map($decoded).asInstanceOf[$rtype]")
25
26 }.unzip
27
28 q"""
29 new CaseClassMapConverter[$tpe] {
30 def toMap(t: $tpe): Map[String,Any] = Map(..$toParams)
31 def fromMap(map: Map[String,Any]): $tpe = $companion(..$fromParams)
32 }
33 """
34 }
35 }
First, trait CaseClassMapConverter[C] is a typeclass that represents the behavior functions of type C data, toMap and fromMap. At the same time, we can see that the Macro definition implicit def Materializer[C] is implicit and generic, and the operation result type is CaseClassMapConverter[C]. It can be inferred from this that the Macro definition can generate CaseClassMapConverter[C] instances through the Macro implementation function, and C can be any case class type. An example of the implicit parameter CaseClassMapConverter[C] required by the functions ccToMap and mapTocc is provided by this Macro implementation function. Note that we can only use WeakTypeTag to get information about type parameter C. When using quasiquotes, we usually put the original code in q brackets. Call AST variables in q brackets with a $prefix (called unquote). For the operation of type tpe, you can refer to scala.reflect api. The demo call code is in ConverterDemo.scala in the demo directory:
1 import CaseClassMapConverter._
2 object ConvertDemo extends App {
3
4 def ccToMap[C: CaseClassMapConverter](c: C): Map[String,Any] =
5 implicitly[CaseClassMapConverter[C]].toMap(c)
6
7 case class Person(name: String, age: Int)
8 case class Car(make: String, year: Int, manu: String)
9
10 val civic = Car("Civic",2016,"Honda")
11 //println(ccToMap[Person](Person("john",18)))
12 //println(ccToMap[Car](civic))
13
14 def mapTocc[C: CaseClassMapConverter](m: Map[String,Any]) =
15 implicitly[CaseClassMapConverter[C]].fromMap(m)
16
17 val mapJohn = ccToMap[Person](Person("john",18))
18 val mapCivic = ccToMap[Car](civic)
19 println(mapTocc[Person](mapJohn))
20 println(mapTocc[Car](mapCivic))
21
22 }
In the implicit Macros example above, some quasiquote statements (q"xxx") are quoted. Quasiquote is an important part of Scala Macros. It mainly replaces the reify function of reflection api. It has more powerful, convenient and flexible AST processing function. Scala Def Macros also provides Extractor Macros, which combines Scala String Interpolation with pattern matching to provide extractor object generation of compile time. Specific uses of Extractor Macros are as follows:
1 import ExtractorMicros._
2 val fname = "William"
3 val lname = "Wang"
4 val someuser = usr"$fname,$lname" //new FreeUser("William","Wang")
5
6 someuser match {
7 case usr"$first,$last" => println(s"hello $first $last")
8 }
In the example above, the usr of usr "??" is a pattern and extractor object. Unlike the usual string interpolation, usr is not a method, but an object. This is because unapply in pattern matching must be in an extractor object, so usr is an object. We know that an object plus its application can be called as a method. That is to say, if apply is implemented in usr object, usr(???) can be used as method, as follows:
1 implicit class UserInterpolate(sc: StringContext) {
2 object usr {
3 def apply(args: String*): Any = macro UserMacros.appl
4 def unapply(u: User): Any = macro UserMacros.uapl
5 }
6 }
Def Macros generates apply and unapply automatically during compilation, which correspond to function calls respectively:
val someuser = usr"$fname,$lname"
case usr"$first,$last" => println(s"hello $first $last")
The following is the implementation of macro appl:
1 def appl(c: Context)(args: c.Tree*) = {
2 import c.universe._
3 val arglist = args.toList
4 q"new FreeUser(..$arglist)"
5 }
The construction of an AST is mainly realized by q"new FreeUser(arg1,arg2)". The implementation of macro uapl is relatively complex, and the application of quasiquote will be more in-depth. Firstly, the number and name of the parameters of the primary constructor of the type are determined. Then, the corresponding sub-AST is decomposed by the pattern matching of quasiquote, and the final complete AST is formed by recombination.
1 def uapl(c: Context)(u: c.Tree) = {
2 import c.universe._
3 val params = u.tpe.members.collectFirst {
4 case m: MethodSymbol if m.isPrimaryConstructor => m.asMethod
5 }.get.paramLists.head.map {p => p.asTerm.name.toString}
6
7 val (qget,qdef) = params.length match {
8 case len if len == 0 =>
9 (List(q""),List(q""))
10 case len if len == 1 =>
11 val pn = TermName(params.head)
12 (List(q"def get = u.$pn"),List(q""))
13 case _ =>
14 val defs = List(q"def _1 = x",q"def _2 = x",q"def _3 = x",q"def _4 = x")
15 val qdefs = (params zip defs).collect {
16 case (p,d) =>
17 val q"def $mname = $mbody" = d
18 val pn = TermName(p)
19 q"def $mname = u.$pn"
20 }
21 (List(q"def get = this"),qdefs)
22 }
23
24 q"""
25 new {
26 class Matcher(u: User) {
27 def isEmpty = false
28 ..$qget
29 ..$qdef
30 }
31 def unapply(u: User) = new Matcher(u)
32 }.unapply($u)
33 """
34 }
35 }
Most of the previous code is to form a List[Tree] qget and qdef, and finally to combine a complete quasiquote Q "," "new {...}"".
The complete Macro implementation source code is as follows:
1 trait User {
2 val fname: String
3 val lname: String
4 }
5
6 class FreeUser(val fname: String, val lname: String) extends User {
7 val i = 10
8 def f = 1 + 2
9 }
10 class PremiumUser(val name: String, val gender: Char, val vipnum: String) //extends User
11
12 object ExtractorMicros {
13 implicit class UserInterpolate(sc: StringContext) {
14 object usr {
15 def apply(args: String*): Any = macro UserMacros.appl
16 def unapply(u: User): Any = macro UserMacros.uapl
17 }
18 }
19 }
20 object UserMacros {
21 def appl(c: Context)(args: c.Tree*) = {
22 import c.universe._
23 val arglist = args.toList
24 q"new FreeUser(..$arglist)"
25 }
26 def uapl(c: Context)(u: c.Tree) = {
27 import c.universe._
28 val params = u.tpe.members.collectFirst {
29 case m: MethodSymbol if m.isPrimaryConstructor => m.asMethod
30 }.get.paramLists.head.map {p => p.asTerm.name.toString}
31
32 val (qget,qdef) = params.length match {
33 case len if len == 0 =>
34 (List(q""),List(q""))
35 case len if len == 1 =>
36 val pn = TermName(params.head)
37 (List(q"def get = u.$pn"),List(q""))
38 case _ =>
39 val defs = List(q"def _1 = x",q"def _2 = x",q"def _3 = x",q"def _4 = x")
40 val qdefs = (params zip defs).collect {
41 case (p,d) =>
42 val q"def $mname = $mbody" = d
43 val pn = TermName(p)
44 q"def $mname = u.$pn"
45 }
46 (List(q"def get = this"),qdefs)
47 }
48
49 q"""
50 new {
51 class Matcher(u: User) {
52 def isEmpty = false
53 ..$qget
54 ..$qdef
55 }
56 def unapply(u: User) = new Matcher(u)
57 }.unapply($u)
58 """
59 }
60 }
Call the demonstration code:
1 object Test extends App {
2 import ExtractorMicros._
3 val fname = "William"
4 val lname = "Wang"
5 val someuser = usr"$fname,$lname" //new FreeUser("William","Wang")
6
7 someuser match {
8 case usr"$first,$last" => println(s"hello $first $last")
9 }
10 }
Macros Annotation is an important functional part of Def Macro. Annotating a target, including types, objects, methods, etc., means extending, modifying or even completely replacing source code at compilation time. For example, the method annotation shown below: suppose we have the following two methods:
1 def testMethod[T]: Double = {
2 val x = 2.0 + 2.0
3 Math.pow(x, x)
4 }
5
6 def testMethodWithArgs(x: Double, y: Double) = {
7 val z = x + y
8 Math.pow(z,z)
9 }
If I want to test the time they take to run, I can set the start time before the internal code of the two methods, and then intercept the completion time after the code. The completion time-start time is the time required for the operation, as follows:
1 def testMethod[T]: Double = {
2 val start = System.nanoTime()
3
4 val x = 2.0 + 2.0
5 Math.pow(x, x)
6
7 val end = System.nanoTime()
8 println(s"elapsed time is: ${end - start}")
9 }
We want to expand this approach through annotations: by retaining the original code and adding a few lines of code inside the method. Let's see how the purpose of this annotation is achieved:
1 def impl(c: Context)(annottees: c.Tree*): c.Tree = {
2 import c.universe._
3
4 annottees.head match {
5 case q"$mods def $mname[..$tpes](...$args): $rettpe = { ..$stats }" => {
6 q"""
7 $mods def $mname[..$tpes](...$args): $rettpe = {
8 val start = System.nanoTime()
9 val result = {..$stats}
10 val end = System.nanoTime()
11 println(${mname.toString} + " elapsed time in nano second = " + (end-start).toString())
12 result
13 }
14 """
15 }
16 case _ => c.abort(c.enclosingPosition, "Incorrect method signature!")
17 }
As you can see, we also use quasiquote to split the annotated method, and then use quasiquote to reproduce the combination method. In the process of reorganization, time interception and printing codes are added. The following line is a typical AST pattern descension:
case q"$mods def $mname[..$tpes](...$args): $rettpe = { ..$stats }" => {...}
In this way, the goals are broken down into the most basic parts of the reorganization needs. The source code for Macro Annotation is as follows:
1 import scala.annotation.StaticAnnotation
2 import scala.language.experimental.macros
3 import scala.reflect.macros.blackbox.Context
4
5 class Benchmark extends StaticAnnotation {
6 def macroTransform(annottees: Any*): Any = macro Benchmark.impl
7 }
8 object Benchmark {
9 def impl(c: Context)(annottees: c.Tree*): c.Tree = {
10 import c.universe._
11
12 annottees.head match {
13 case q"$mods def $mname[..$tpes](...$args): $rettpe = { ..$stats }" => {
14 q"""
15 $mods def $mname[..$tpes](...$args): $rettpe = {
16 val start = System.nanoTime()
17 val result = {..$stats}
18 val end = System.nanoTime()
19 println(${mname.toString} + " elapsed time in nano second = " + (end-start).toString())
20 result
21 }
22 """
23 }
24 case _ => c.abort(c.enclosingPosition, "Incorrect method signature!")
25 }
26
27 }
28 }
The annotated invocation demo code is as follows:
1 object annotMethodDemo extends App {
2
3 @Benchmark
4 def testMethod[T]: Double = {
5 //val start = System.nanoTime()
6
7 val x = 2.0 + 2.0
8 Math.pow(x, x)
9
10 //val end = System.nanoTime()
11 //println(s"elapsed time is: ${end - start}")
12 }
13 @Benchmark
14 def testMethodWithArgs(x: Double, y: Double) = {
15 val z = x + y
16 Math.pow(z,z)
17 }
18
19 testMethod[String]
20 testMethodWithArgs(2.0,3.0)
21
22
23 }
One thing worth noting is that Macro extensions occur when method calls are encountered in compilation, while annotation target extensions occur when method declarations are made earlier. Let's look again at the example of annotated class:
1 import scala.annotation.StaticAnnotation
2 import scala.language.experimental.macros
3 import scala.reflect.macros.blackbox.Context
4
5 class TalkingAnimal(val voice: String) extends StaticAnnotation {
6 def macroTransform(annottees: Any*): Any = macro TalkingAnimal.implAnnot
7 }
8
9 object TalkingAnimal {
10 def implAnnot(c: Context)(annottees: c.Tree*): c.Tree = {
11 import c.universe._
12
13 annottees.head match {
14 case q"$mods class $cname[..$tparams] $ctorMods(..$params) extends Animal with ..$parents {$self => ..$stats}" =>
15 val voice = c.prefix.tree match {
16 case q"new TalkingAnimal($sound)" => c.eval[String](c.Expr(sound))
17 case _ =>
18 c.abort(c.enclosingPosition,
19 "TalkingAnimal must provide voice sample!")
20 }
21 val animalType = cname.toString()
22 q"""
23 $mods class $cname(..$params) extends Animal {
24 ..$stats
25 def sayHello: Unit =
26 println("Hello, I'm a " + $animalType + " and my name is " + name + " " + $voice + "...")
27 }
28 """
29 case _ =>
30 c.abort(c.enclosingPosition,
31 "Annotation TalkingAnimal only apply to Animal inherited!")
32 }
33 }
34 }
We see that AST mode splitting is also done through quasiquote:
case q"$mods class $cname[..$tparams] $ctorMods(..$params) extends Animal with ..$parents {$self => ..$stats}" =>
And then recombine. Specific use demonstrations are as follows:
1 object AnnotClassDemo extends App { 2 trait Animal { 3 val name: String 4 } 5 @TalkingAnimal("wangwang") 6 case class Dog(val name: String) extends Animal 7 8 @TalkingAnimal("miaomiao") 9 case class Cat(val name: String) extends Animal 10 11 //@TalkingAnimal("") 12 //case class Carrot(val name: String) 13 //Error:(12,2) Annotation TalkingAnimal only apply to Animal inherited! @TalingAnimal 14 Dog("Goldy").sayHello 15 Cat("Kitty").sayHello 16 17 }
The results are as follows:
Hello, I'm a Dog and my name is Goldy wangwang...
Hello, I'm a Cat and my name is Kitty miaomiao...
Process finished with exit code 0