Edgar Gonçalves

Software Engineer, PhD student;
#lisp, #web-engineering;

Clojure and TDD for the lazy ones

I’m fond of TDD, and Clojure makes it fun to do. Alas, often I find myself with clojure files without tests, and I feel the urge to write some tests to it. Enter clojure-test-mode.el (part of Phil Hagelberg’s excellent clojure-mode @ https://github.com/technomancy/clojure-mode/, for Emacs). This addition makes it fun to write and run tests in clojure (almost like a game, we’ll thrive to make all tests pass!). So what’s wrong with this?

My problem is one of lazyness. If I’m on a source code file, and I want to switch to the test for it, currently I can only go to the corresponding test file —– then look for the appropriate test. Moreover, if the test isn’t already there, I have to create one (which, with long test files, may incurr on a mistaken duplication of a test…). So I wrote a little something to help me out. Picking on the already defined (clojure-jump-to-test), my function creates the required skeleton on the test file, if it isn’t already there. Then it places the cursor on the first form inside the test we want to (named in the format test-MY-FUNCTION-NAME). If this test isn’t there, a “deftest” form is created. See, much better, if you need to do this dozens of times!

(defun clojure-jump-to-function-test ()
  (interactive)
  (flet ((maybe-create-test-namespace
    (namespace)
    (unless (clojure-find-package)
      (goto-char (point-min))
      (insert (format "(ns %s\n  (:use [%s] [clojure.test]))\n\n"
              (replace-regexp-in-string "\\(\\.\\)[^\\.]+$" ".test." namespace nil nil 1)
              namespace)))))
  (let* ((fn (which-function))
   (fn-name (if (listp fn) (first fn) fn))
   (namespace (clojure-find-package)))
    (clojure-jump-to-test)
    (when fn-name
(goto-char (point-min))
(maybe-create-test-namespace namespace)
(unless (search-forward-regexp (format "(deftest test-%s[ \t\n]*" fn-name) nil t)
  (goto-char (point-max))
  (insert (format "\n\n(deftest test-%s\n  )" fn-name))
  (goto-char (- (point) 1)))))))

Then it’s a matter of using this function. I prefer to replace the shortcut used on clojure-mode to jump to a test, so I place this bit after the function:

(define-key clojure-mode-map (kbd "C-c t") 'clojure-jump-to-function-test)

Happy testing!

Making better use of Leiningen with Lancet

I suppose I won’t say anything new to anyone who has been around Clojure for a while. But did you know you can use the Leiningen configuration file in your project for other than… well, configuring your project? :d

This is something that needs to be clarified – At least I was asked at least a couple of times, and after a quick google, I didn’t find a simple description on the right way to do it. So here it is, one concrete example of how I got fed up of launching more than 3 shell commands each time I needed to perform a stop-compile-redeploy-start cycle. True, I could had chosen the way of the shell script, or made yet another ant build.xml. But I already had one project.el that I really needed – so why increase my project’s heterogeneity/entropy?

What I did was to list the steps I was performing.

  1. Stopping Tomcat (calling a shutdown.shshell script);
  2. Making a jar of my clojure project with lein uberjar;
  3. Calling out ant jar on another project’s dir, to be compiled against the clojure project’s uberjar;
  4. Calling out ant deploy on yet another project, to make a final compilation step against the first two, and to deploy the produced WAR directory to my Tomcat’s container;
  5. Starting again the Tomcat (with the startup.sh shell script).

Let’s take a look: we have a couple of shell script calls, a couple of ant tasks called on external projects, and a leiningen command. Now to put this all together…

The first step is to realize that the project.clj is just a clojure file, that lein calls with load-file. So we may add things to that file other than the standard (defproject ...) form, and those will be evaluated. This is perfect! So now we have to decide how we’re going to call this build command. Lein supports a plugin architecture. A lein plugin is a function foo/foo that shall be called in the command line with lein foo. And if you take a look at the existing plugins you’ll realize they are nothing more than Clojure projects themselves. But since we’re making a cozy little lein command, useful for our setup only (at least at a first sight), maybe we don’t need the entire folder structure of a clojure project. We can take advantage of the fact that project.clj is already being called to define our project, before any command is performed.

So now how will we do each of the mentioned step? For calling out uberjar, and looking at its source we see it’s a simple function (leiningen.uberjar/uberjar) that accepts a mandatory project variable. What exactly is this? Take a minute to look around the leiningen.core to realize this is an internal variable, so we’ll feed it to the uberjar function.

What about the other steps? If only there were a form of calling ant tasks, from java… Wait, that’s it! Ant has, in fact, a nice Java API! Unfortunately for Ant, it’s usage is rather cumbersome. Fortunately for us, Clojure is a great language, Stuart Halloway thinks it too – so much that he created Lancet – and leiningen is built on top of it! You can grasp Lancet by reading it’s source, but the really fun way is to read Hallowway’s “Programming Clojure”, where Lancet is created step by step. The gist of it, for our present issue, is that you can call ant tasks as clojure functions easily. Just like ant, to call other ant build-files' targets, and exec to perform a shell command (take a look at some of the built-in options, and don’t forget you may add third-party libraries with ant tasks!).

In the next example I illustrate how I added the mentioned steps to my lein setup:

You see how I added up a couple of helper functions, and created a deploy that shall do all I need. Yes, this isn’t perfect: if, e.g., the first ant call fails, the workflow isn’t stopped. But you can always send an INTerrupting signal to the terminal and kill it all. And you can always write more specific lein tasks, detect error codes, etc. That simply goes outside the scope of this post – quick hacks, great improvements!

One important remark: leiningen’s plugins apparently weren’t made to be developed interactively. I mean, you write them, and you run them in a shell. Otherwise, you’ll have to go through some lengths to include the leiningen’s jar into your project, just to let slime (or your editor) know its namespaces (as well as lancet’s). But again, for a small hack, you’ll be fine without it. At least I was. But if and when I find myself doing more of such scripts, I just may scribble an emacs command to fire up a swank server with lein on its classpath.

EDIT: I spoke too soon. I just updated to the new Leiningen 1.3.0, and suddenly my project.clj stopped being fully loaded. Discoveries:

  • Now we have to place all custom boot scripts separated from your (defproject ...) form – in a project-root/leiningen/my-custom-command.clj (for a command named my-custom-command inside a equally named namespace). Still don’t know if I’m doing something wrong, but it’s probably for the best – give each command its own place and keep everything tidy.
  • I discovered much of what I’ve just explained is nicely described on http://github.com/technomancy/leiningen/blob/master/PLUGINS.md, but you may find a specific example like the one I just gave you more useful.

Learning to switch from (a) common lisp to idiomatic clojure

Sometimes you just want to show your new toys to your friends. This particular time, the new toy was Clojure. And the friend in question happened to have a (albeit reduced) Lisp background. So he was about to start a new function to find out if some arguments were all in a specific list, his train of thought was something like this:

Return true if some of those arguments are not in the [:read :written :deleted] sequence.

Having just read about the arrays being useful as functions, he tried to exploit that. He skimmed the documentation on the clojure site, he found out there is no contains nor a is-in, but there was a some. so he wrote:

(defn only-right-args? [args]
  "Only the keywords :read :written and :deleted will be valid."
  (not (some (fn [x] (not ([:read :written :deleted] x)) args))))

Now this is interesting. We see a literal approach here. Even without evaluating (and without spotting the obvious reason for it to fail), I introduced him to the anonymous functions syntactic sugar. And so he wrote:

(defn only-right-args? [args]
  "Only the keywords :read :written and :deleted will be valid."
  (not (some #(not ([:read :written :deleted] %)) args)))

Not bad. But reading “not-some-not” isn’t pleasant. So I told him about every?, that he somehow missed in the documentation. And things got a bit cleaner:

(defn only-right-args? [args]
  "Only the keywords :read :written and :deleted will be valid."
  (every? #([:read :written :deleted] %) args))

But in terms of redundancy, I explained that he was making an anonymous function with one argument with the single purpose of applying another function of one argument, and that – being in a language with functions as first-class-citizens – he might as well pass that function. He found it weird (afterall, seeing an array isn’t immediately obvious that it’s also a function), but after a (very short) while he started preferring this:

(defn only-right-args? [args]
  "Only the keywords :read :written and :deleted will be valid."
  (every? [:read :written :deleted] args))

Great – much simpler than the first attempt! Now it’s time to feed this to the repl, and get – suprise! – disappointed… That’s right, an array can be used as a function, but one that receives an index as argument. After reading about this, he thought of using a map, but since maps would need to have values related to the keys, he would have to do something like {:read true, :written true, :deleted true}, or even (zipmap [:read :written :deleted] [true true true]), but neither was simple enough. And at this point I mentioned the concept of Sets. So he quickly fixed his code to this:

(defn only-right-args? [args]
  "Only the keywords :read :written and :deleted will be valid."
  (every? #{:read :written :deleted} args))

A quick test showed him that this code worked, as well as being easy on the eye:

user> (only-right-args? [:read :written])
true
user> (only-right-args? [:read :blacksheep :written])
false

This was fun. Some valuable lessons were learnt, specially on how it’s generally better to give in, when talking about Clojure’s idiomatic forms taking over the more traditional “show-me-how-to-open-and-close-parenthesis-and-ill-code-whatever-i-need-to-do-it” philosophy!

Now, a question for you, gentle reader – What would you improve in the latest function definition? Is there a better/faster/cleaner way to do it in Clojure?

Faster swap of Clojure namespaces in Emacs buffers

Last night I got fed up with not having a working way to swap the current namespace in a Clojure SLIME session (on Emacs) without switching to a REPL and evaluating the form (in-ns 'desired.namespace).

Round 1: making it work

I asked around about this, and got the impression that I was witnessing the “proper” behavior. I then coded a simple emacs hack to do just that:

It was only after publishing that snippet that swank-clojure’s author shed some light to me about the almost equivalent C-c M-p emacs command. It’s certainly handier to use that instead of going back and forth between the file and the REPL. But now I see how much more useful my function is. I find myself using it almost every time, since it doesn’t require any input from me.

But it wasn’t perfect, as I quickly could see. What I really wanted was a mix, a command that could do both interactions variations.

Round 2: improving and generalizing it

So I quickly made use of the interactive emacs function, and reused swank-clojure’s function. The outcome is the following:

I find this function to work pretty fast, but feel free to suggest improvements.

EDIT: Jürgen Hötzel suggested that I could use the clojure-find-package function, and I did so, erasing some redundant code. Thanks! Here’s the revised function:

Aftermatch: the best shortcut

So now the only thing I’m left with is a decision – what keyboard shortcut should I assign this function to? What makes sense to me, is to replace the C-c ~ (since it doesn’t make sense in Clojure) but somehow I’m not managing to do it. So for now I’ve put it under C-x 4 P (I know it’s a global keyboard-space, but it’s a reminder that I need to fix this)

Any suggestions?

Using Clojure with AspectJ's around advices

Context

I wanted to make a call chain, using clojure. And I wanted to track calls to methods of a given Class (the entire service layer, in this case). Using plain Java, the way to tackle this was to use AspectJ, insert a pointcut and an advice, like this:

public aspect MyAspect
{
    pointcut allServiceCalls(): execution(* server.ServiceImpl*(..));
    before(): allServiceCalls() {
        System.out.println(thisJoinPointStaticPart.getSignature().getName());   
    }
}

To use a clojure call to register this call would be trivial, Just use the method I wrote about in the last post, and place the call where the System.out.println is. Simple, right? Let’s move on to where I spend a bit more time figuring my way out: around advices.

The Problem

An around advice, in AspectJ, is much like a before or after advice, except that the advice is responsible for calling proceed();. In order to compute the depth, the easiest way would be to use a dynamically bound variable, update it before calling proceed, and reverting it to its previous value afterwards. In languages like Common Lisp this would be trivial: call defvar to specify a dynamic variable, and use the let macro wrapping the call itself, updating the dynamic variable. But Java won’t allow this behavior, as the proceed(); call is a blackbox, enclosed, and has access only to its parameters, its instance and its parents'. Not to variables that were on the same frame of its call.

So I thought clojure should help, bringing a bit of lisp magic. As it turns out, the binding keyword, used like let, allows us to change dynamic variables, thread-scoped. Perfect! So we just make a before and an after method, and we’re good to go. Wait, but then I can’t wrap them on the clojure side, they’re on different frames, so the binding won’t be able to wrap the proceed(); call… We must definitely use the around advice! But in aspectJ, the proceed is a special keyword, so we won’t be able to use it directly in Clojure. Sure, we could dig around in the aspectJ’s more comprehensive (read: longer, uglier) syntax and use a more primitive way to make the call.

Ahh, if only I could pass a lambda with that computation as an argument to the clojure code… Wait, that’s it! Though not really as practical nor as pretty as Lisp’s lambda notation, Java does allow us to capture behaviour: just make an inline class instance sporting a method to perform the computation! In fact, Java already contains java.lang.Runnable to do just that. Let’s use it, then.

The Solution

Let’s create a subclass of Runnable, so that we may store the computed value and retrieve it later:

package test;
public abstract class AroundWorker implements Runnable {
    private Object returnValue;
    public Object getReturnValue() {
        return returnValue;
    }
    public void setReturnValue(Object returnValue) {
        this.returnValue = returnValue;
    }
}

Now we can use this worker to encapsulate the call to proceed, like this:

public aspect MyAspect
{
    pointcut allServiceCalls(): execution(* server.ServiceImpl*(..));
    Object around(): allServiceCalls() {
        test.AroundWorker worker = new test.AroundWorker() {
            public void run() {
                    this.setReturnValue(proceed());
            }
        };
        my.namespace.markCall(worker, 
            thisJoinPointStaticPart.getSignature().getName());  
        return worker.getReturnValue();
    }
}

Perfect! The variable worker is just like a lambda definition. Yes, I’d rather use (lambda () (proceed)), but it’s Java we’re talking about. I’m glad just for being able to do what I want :)

Now the Clojure part of the code is simple, I’ll just write it down to illustrate how bindings can be used to help us on this problem:

(ns my.namespace
(:gen-class
    :name my.namespace
    :methods [#^{:static true} [markCall [Runnable String] void]]))
(def *current-depth-level* 0)
(defn mark-call
    "Registers a call event, printing its depth level."
    [runnable fn-name]
    (binding [*current-depth-level* (+ *current-depth-level* 1)]
        (println (format "=> current-depth-level %s: %s" 
                            *current-depth-level*
                            fn-name))
        (.run runnable)))
(defn -markCall [& args] (apply mark-call args))

There you go. Run it, and you should see your calls being identified with the proper call depth level. Notice again how I make a wrapper function, exported to the Java world, the encapsulates a clojure one. This allows me to fire up a swank server on the clojure side, and making changes to mark-call at will, without having to recompile the uberjar and reloading/recompiling the java classes. Priceless! :)

The future

It’s completely out of my scope and time-frame for my current project, but it would be great to be able to do this kind of advices directly, without having to resort to aspectJ. If anyone knows about ongoing projects to do this on clojure, please, do leave a comment, I’d be glad to give it a test!

Posterous testdrive and java.callClojureFns()

Welcome to my new posterous!

I’m fairly fed-up with my blogger experience. So much that I feel it’s time to move on to a more ubiquitous platform – posterous looks a great candidate. So this can be a farewell to blogger!

Let’s test drive this thing, and I take this opportunity to write how you’re able to call a Clojure fn from your Java code. (Yes, this has been seen on other places, but allow me to gather this bit of info here, for future reference).

Java side

You’ll need to import the package with the exact name of the namespace you intend to call from. For instance, in my current project, I have something like this:

// Foo.java 
    import offline.profile;
    public class Foo {
        public static void main(String[] args) throws Exception {
             // Call it!
        profile.markCall("function A", new String[] {"Argument 1", "Argument 2"});
        profile.printEvents();
        }
    }

And that’s it!

Clojure side

Now you’ll have to specify exactly what functions are going to be available on the Java side. You do this inside the ns statement, by specifying the proper class name (offline.profile, in this case) and by telling clojure to generate a class with some specific public functions:

(ns offline.profile
      (:gen-class
       :name offline.profile
       :methods [#^{:static true} [markCall [String "[Ljava.lang.String;"] void]
                      #^{:static true} [printEvents [] void]]))

Now if you pay attention to the signature, I chose to make static methods, but I don’t need to. Also, the markCall method receives a String and an array of Strings, but since the ns macro doesn’t evaluate its arguments, we have to pass the equivalent String of the type declaration. I believe that you can also use the gen-class isolated from the ns macro, without this problem, but I haven’t tested it so far.

After this, you’ll also have to specify your functions with a prefix (defaults to “–”). So in this case, you’d have these:

(defn -printEvents
      "A Java-callable wrapper around the 'print-events' fn."
      []
      (print-events))
    (defn -markCall
      "Register a function call event."
      [fn-name & args]
      (mark-event (make-event "CALL" (list fn-name args))))

Note that by wrapping your clojure code in an java-callable method, you’ll be able to redefine the former (e.g., (print-events) ) without having to reload your java program. How do you do that? Read along for a useful bit.

Use your REPL

There may be other ways to do this, but I’ll keep this information related to my development environment – Emacs, SLIME, swank-clojure. The idea is to fire up a swank server inside your application, so that you may do a \M-x slime-connect and evaluate, debug, inspect at your own pleasure, even from all your clj files. In the project I’m working on, I have a Tomcat-deployd webapp that I can inspect and manipulate dynamically (with some “minor” restrictions, I may have to blog about eventually). And this is, how do you say it, great! :) One word of advice: (at least) the latest dev version of aquamacs, based on emacs24, doesn’t like having the swank-server closed while slime is trying to access it. So be gentle with it, quit your slime first, shutdown tomcat (or your app) afterwards. Here’s the clj file that loads up a swank-server (call it in java like any other clojure-exported function):

(ns offline.Embed
      (:require [swank.swank :as swank])
      (:gen-class
       :methods [#^{:static true} [startSwank [] void]]))
    (defn -startSwank []
      (swank/start-repl))

Epilogue

Let’s see if this posterous thing can convince me. And, almost more importantly, whether it can convince you!