26 August, 2014 by Denise << scala >>

Option and flatMap

What is an Option anyway?

One of the things that confused me when I first started learning about the Option type in Scala was that I couldn't understand why it was possible to call flatMap and flatten on an Option type.

It turns out that Option[T] can be considered a special type of collection with two possible values - either a collection with zero elements, or a collection with exactly one value of type T.

It's possible in Scala to convert an Option into a list, by calling the toList method on it, which yields some interesting results:

scala> val opt = Some("bunny")
opt: Some[String] = Some(bunny)
scala> opt.toList
res5: List[String] = List(bunny)

scala> val noBunny = None
noBunny: None.type = None
scala> noBunny.toList
res6: List[Nothing] = List()

Indeed, we can see that the None value yields the empty list, and that the Some value yields a list with one element.

So what does that mean?

Now we know that Option can be treated as a collection, it means we get to use them in a functional way, and all the funtional operators like map and flatMap apply.

One particular use case I came across was where I wanted to do some processing on a List[User] where User had an Option[String] type as its username. I didn't want to do the processing if the username value was None. Here's the code:

As you can see, I'm using flatMap to convert any None values to the empty List and effectively remove them from my processing.

I have also included the flatMap and map version converted to using for comprehensions which you may find a little easier to read, although to the Scala beginner they often look a little magical.

18 August, 2014 by Denise << java, play-framework, api >>

Screen Scraping Secure Pages

Recently I needed an API for some data I wanted to use in an app. The data was behind a secure login with no API exposed and I decided to screen scrape. Unfortunately, screen scraping behind secure pages is a little tricker and so to get around it, I wrote some code to hit the login page, parse the cookies and authenticity token, and then post to the form url with the above information and the user provided login credentials.

The code looks like this:

Note that you should only do this kind of scraping if you're the owner of the site and can't or don't have time to implement a proper API. Additionally, this code is really brittle and any structural changes to the DOM will probably cause your screen scraping to fail.

One (reasonable) use case for implementing something like this might be where you have an old website which for some reason you can't update to build a real API but you need one to access the information it provides.

Another thing to consider is that given we're passing the username and password straight through to our code, if you decide to expose the code as a public API it should be over HTTPS only.

09 July, 2014 by Denise << scala, functional-programming >>

Interesting Scala Compiler Features

I've learnt a number of interesting things about the Scala compiler recently, the first of which is the @annotation.tailrec annotation - if you add this to a method which you expect to be tail call optimised, and the compiler can't perform the optimisation, then it will generate an error. Usage of it looks like this:

  def factorial(n: Int): Int = {
    def go(n: Int, acc: Int): Int =
      if (n <= 0) acc
      else go(n-1, n*acc)
    go(n, 1)

The other one I came across when doing an exercise with currying was where I had written the following two functions:

  def curry1[A,B,C](f: (A, B) => C): A => (B => C) =
    (a: A) => f(a, _)

  def curry2[A,B,C](f: (A, B) => C): A => (B => C) =
    (a: A) => (b: B) => f(a, b)

I wanted to prove that the two functions were equivalent. After some research I found that I could pass a flag to the Scala compiler when compiling my class like this:

scalac -Xprint:typer Currying.scala

This command prints out the abstract syntax tree of the code after the compiler phase where types are assigned has been run. This is what was printed out:

[[syntax trees at end of                 refchecks]] // Currying.scala
package  {
  object Currying extends scala.AnyRef {
    def (): Currying.type = {
    def curry1[A, B, C](f: (A, B) => C): A => (B => C) = ((a: A) => ((x$1: B) => f.apply(a, x$1)));
    def curry2[A, B, C](f: (A, B) => C): A => (B => C) = ((a: A) => ((b: B) => f.apply(a, b)))

So yes, they are equivalent!

Then, I started wondering how many phases there were in the Scala compiler, and found that the command:

scalac -Xshow-phases

will list each phase. I am using version 2.11.1 of the compiler and my output looked like this:

    phase name  id  description
    ----------  --  -----------
        parser   1  parse source into ASTs, perform simple desugaring
         namer   2  resolve names, attach symbols to named trees
packageobjects   3  load package objects
         typer   4  the meat and potatoes: type the trees
        patmat   5  translate match expressions
superaccessors   6  add super accessors in traits and nested classes
    extmethods   7  add extension methods for inline classes
       pickler   8  serialize symbol tables
     refchecks   9  reference/override checking, translate nested objects
       uncurry  10  uncurry, translate function values to anonymous classes
     tailcalls  11  replace tail calls by jumps
    specialize  12  @specialized-driven class and method specialization
 explicitouter  13  this refs to outer pointers
       erasure  14  erase types, add interfaces for traits
   posterasure  15  clean up erased inline classes
      lazyvals  16  allocate bitmaps, translate lazy vals into lazified defs
    lambdalift  17  move nested functions to top level
  constructors  18  move field definitions into constructors
       flatten  19  eliminate inner classes
         mixin  20  mixin composition
       cleanup  21  platform-specific cleanups, generate reflective calls
    delambdafy  22  remove lambdas
         icode  23  generate portable intermediate code
           jvm  24  generate JVM bytecode
      terminal  25  the last phase during a compilation run

Who knew there were so many different phases, and how interesting!

08 July, 2014 by Denise << functional-programming >>

Iterative to Functional Recursion

So I'm working through a book called 'Functional Programming in Scala' and one of the exercises was to implement a recursive method for checking whether an array is sorted. The signature given was:

def isSorted[A](as: Array[A], gt: (A,A) => Boolean): Boolean

My first attempt looked like this:

 def isSorted[A](as: Array[A], gt: (A,A) => Boolean): Boolean = {

    def loop(index: Int): Boolean = {
      if (index >= (as.length - 1)) true
      else if (gt(as(index), as(index + 1))) loop(index + 1)
      else false

but the array indexing was bothering me and I thought there must be a 'more functional' way. So I asked for a code review and was told that using a list might help, I reimplemented it as follows:

      def isSorted2[A](as: Array[A], gt: (A,A) => Boolean): Boolean = {

    def loop(as: Array[A]): Boolean = {
      if (as.length == 1 || as.length == 0) true
      else if (gt(as.head, as.tail.head)) false
      else loop(as.tail)

I think this reads a lot cleaner, and I don't need to keep track of the index in my head anymore. These imperative ways of thinking are hard to break though!

01 July, 2014 by Denise << programming, scala, play-framework >>

CORS and (Scala) Play

What is CORS?

CORS stands for 'Cross Origin Resource Sharing'. It's a way for a website to access resources (e.g. via AJAX calls) that are not on the same domain. This includes the case where only the ports are different, i.e. http://localhost:1111 is not considered to be the same domain as http://localhost:2222. These sorts of requests are forbidden by browsers because of the same origin security policy.

When might you need to deal with CORS?

You'll often need to work around CORS when you have an API serving JSON which you want to call from your fancy Javascript MV* framework application. You might deploy your back end to Heroku or AWS and have your front end code deployed to Cloudfront and serve it from a custom domain. If your back end code doesn't implement CORS then you'll get an HTTP error when your front end tries to call it.

Writing a CORS filter in Play

To implement CORS in the Play Framework, you'll need to create a filter. Mine looks like this, and you can see that all it really does is set a bunch of headers:

package filters

import play.api.mvc._
import scala.concurrent.ExecutionContext.Implicits.global

class CorsFilter extends EssentialFilter {
  def apply(next: EssentialAction) = new EssentialAction {
    def apply(requestHeader: RequestHeader) = {
      next(requestHeader).map { result =>
        result.withHeaders("Access-Control-Allow-Origin" -> "*",
          "Access-Control-Expose-Headers" -> "WWW-Authenticate, Server-Authorization",
          "Access-Control-Allow-Methods" -> "POST, GET, OPTIONS, PUT, DELETE",
          "Access-Control-Allow-Headers" -> "x-requested-with,content-type,Cache-Control,Pragma,Date")

You will also need to reference this filter in your Global.scala class, which you might need to create if it doesn't already exist in your application. Mine looks like this:

import filters.CorsFilter
import play.api.GlobalSettings
import play.api.mvc.WithFilters

object Global extends WithFilters(new CorsFilter) with GlobalSettings


At one point I had an issue where my filter wasn't being executed, no matter what I did, I hooked up the debugger and stepped through the code but for some reason the filter code was never executed. Eventually, I found out that the Global.scala class must be in the default package in your Play application - I had put mine in its own utils package which was why it was never being called.

There is also something that browsers do for more complex requests (like a POST with a MIME type of application/json which is pretty common in these RESTful API style applications) which is called a 'pre-flight request'. This is basically where the browser checks to see whether it is allowed to make the request before performing the full request. It does this by sending an HTTP OPTIONS request, and is something you will also need to handle. The way I did this was to define an additional route in my routes file like this:

OPTIONS        /*all                                            controllers.Application.preflight(all: String)

which is implemented like so in my controller:

  def preflight(all: String) = Action {
    Ok("").withHeaders("Access-Control-Allow-Origin" -> "*",
      "Allow" -> "*",
      "Access-Control-Allow-Methods" -> "POST, GET, PUT, DELETE, OPTIONS",
      "Access-Control-Allow-Headers" -> "Origin, X-Requested-With, Content-Type, Accept, Referrer, User-Agent");


Implementing CORS is often painful and more time consuming than it really should be, so I hope this can help someone else out. Perhaps I should write a Play framework module so this code doesn't have to be re-written every time someone needs to add CORS to their app!


I've created a Play module to implement the above CORS functionality. It's been published to Maven Central. If you have any ideas for improvements or find a bug, please feel free to raise a Github issue or send a pull request through!

Github repo is here: https://github.com/rowdyrabbit/play-cors

Artifact in Maven Central:

15 June, 2014 by Denise << functional-programming >>

For loops and functional programming

I feel like this was probably obvious to everyone except me, but I just realised that the functional programmer's aversion to the for loop is because it has side effects, i.e. in the following code:

for (int i = 0; i < 10; i++) {
    //do stuff

the value of i is modified each time the code loops, which is a side effect.