2013-07-23

Lambda Expressions Backported to Java 7, 6 and 5

Do you want to use lambda expressions already today, but you are forced to use Java and a stable JRE in production? Now that's possible with Retrolambda, which will take bytecode compiled with Java 8 and convert it to run on Java 7, 6 and 5 runtimes, letting you use lambda expressions and method references on those platforms. It won't give you the improved Java 8 Collections API, but fortunately there are multiple alternative libraries which will benefit from lambda expressions.

Behind the Scenes

A couple of days ago in a café it popped into my head to find out whether somebody had made this already, but after speaking into the air, I did it myself over a weekend.

The original plan of copying the classes from OpenJDK didn't work (LambdaMetafactory depends on some package-private classes and would have required modifications), but I figured out a better way to do it without additional runtime dependencies.

Retrolambda uses a Java agent to find out what bytecode LambdaMetafactory generates dynamically, and saves it as class files, after which it replaces the invokedynamic instructions to instantiate those classes directly. It also changes some private synthetic methods to be package-private, so that normal bytecode can access them without method handles.

After the conversion you'll have just a bunch of normal .class files - but with less typing.

P.S. If you hear about experiences of using Retrolambda for Android development, please leave a comment.

Posted by at
Labels:

37 comments:

  1. Lukas Eder23 July, 2013 23:28

    Congrats! That's a pretty awesome idea for early adopters...

    ReplyDelete
  2. Unknown24 July, 2013 11:49

    This comment has been removed by the author.

    ReplyDelete
  3. Unknown24 July, 2013 11:51

    Hi, i'm Sergey, maintainer of HoloEverywhere library for Android.
    Dalvik VM works nicely with retrolambda :)
    Many thanks for library and expirence.

    I use next code for testing: http://pastebin.com/MrXUPn5B
    Screenshot from Android Debug Bridge: http://i.imgur.com/ram0ecB.png

    ReplyDelete
  4. Kevin Parent25 July, 2013 04:32

    This comment has been removed by the author.

    ReplyDelete
  5. Kevin Parent25 July, 2013 04:33

    So it basically let you use lambda expressiona in development for JRE5+ project using JDK8 if I understand ?

    So how is the bytecode looking ? Is it the same as if you were using a a single method interface and an anonymous class instance to simulate lambdas ?

    ReplyDelete
  6. Esko Luontola25 July, 2013 11:00

    Yes, it's pretty much the same as what an anonymous class would look like. There are a couple of wrapper methods causing indirection (so that future JVMs can in the improve the lambda implementation), but the JIT will easily inline them.

    One difference over anonymous classes is that there is an optimization (as is also in JDK 8) that if the lambda doesn't capture values from its enclosing scope, only one instance of the class is ever instantiated (it's stored in a static final field). This optimization will be included in Retrolambda 1.1.0 which I'll release later today or tomorrow.

    ReplyDelete
  7. Kevin Parent25 July, 2013 12:08

    Thanks for the explanations, I will definitively give it a try !

    ReplyDelete
    Replies
    1. Unknown26 July, 2013 02:08

      I've updated my comment in response to your Stack Overflow question about a backport to Java 7. I really didn't believe it could be done so efficiently!

      Delete
    2. Kevin Parent26 July, 2013 09:25

      Cool. Im downloading JDK8 now :)

      Delete
  8. raiph mellor25 July, 2013 18:58

    This comment has been removed by the author.

    ReplyDelete
  9. raiph mellor25 July, 2013 19:05

    (This is basically a repeat of a reddit comment in case this is a better place for you to catch it.)

    I'm impressed across the board -- concept, approach, code, explanation, and dealing calmly with redditors that missed the point. :)

    I note that you've spent some time with Scala and are now in to Clojure. Have you heard of Perl 6, which is, very loosely speaking, in the same sort of space (as compared to the older generation of dynamic langs such as python, ruby, and the original perl)?

    All of this leads me to:

    Perhaps you would be willing to visit the IRC channel #perl6 on freenode [1] and chat for a while? A guy called jnthn is finishing up an initial port of Perl 6 to the JVM [2] and is digging in to java interop, multicores, optimizations, dalvik, and so on. I know they could do with pointers from someone with the level of JVM expertise you must have and I bet he'd enjoy a chat with you even if only for 15 minutes. Here's hoping...

    [1] https://kiwiirc.com/client/irc.freenode.net/perl6?nick=orfjackal
    [2] http://6guts.wordpress.com/2013/07/15/rakudo-jvm-news-more-tests-plus-thread-and-promise-prototypes/

    ReplyDelete
  10. Esko Luontola25 July, 2013 22:05

    I'm not using Perl, so I haven't been following the news surrounding it. (I mainly use Ruby and a little Bash for scripting.)

    Sure, I can chat for a while and try to help what I can. But regarding JVM language development there should be much more knowledgeable people at https://groups.google.com/forum/#!forum/jvm-languages

    ReplyDelete
  11. raiph mellor26 July, 2013 01:25

    Esko, I just saw the exchange in the public log [1]. Thank you. :)

    Fyi, two leading P6ers live in Sweden and work at http://edument.se/english/ including jnthn. I've already seen enough of your work and personality to feel confident they'd be interested in you, as an employee or in their network (aiui their network is not limited to folk living in Sweden). If you're interested, I suggest the same approach, i.e. point jnthn (or masak) at this blog post, mention that I've left two comments, and go from there.

    jnthn, masak, check out http://lets-code.orfjackal.net/

    [1] http://irclog.perlgeek.de/perl6/2013-07-25#i_7370179

    ReplyDelete
  12. dljava27 July, 2013 20:18

    Thanks Esko, my concern is with performance. Have you benchmarked it between a Java 6 and 8 VMs?

    ReplyDelete
  13. Esko Luontola27 July, 2013 23:58

    Nope. You're welcome to benchmark it yourselves (*insert here the usual warnings about microbenchmarks* [1]). Please write an article and post a link here.

    [1] http://stackoverflow.com/questions/504103/how-do-i-write-a-correct-micro-benchmark-in-java https://code.google.com/p/caliper/wiki/JavaMicrobenchmarks

    ReplyDelete
    Replies
    1. Esko Luontola28 July, 2013 00:06

      I'm guessing that the performance will be the same as with the original lambda expressions (or slightly better because of removing the indirection of method handles). The most probable cause of any performance difference is that the older JVMs don't do as many optimizations as the newer JVMs. So make sure to run all benchmarks using the same JVM version.

      Delete
    2. Anonymous04 February, 2015 14:11

      I agree with you.

      Delete
    3. Anonymous04 February, 2015 14:11

      This comment has been removed by the author.

      Delete
  14. Unknown05 September, 2013 07:25

    That's a really Nice Development. Would like to try it out.

    ReplyDelete
  15. Tomas Brandalik26 November, 2013 17:21

    Hi,
    this really looks helpful, but I can't figure out why it doesn't work for me. Am I doing some basic mistake?
    I'm using jdk8-b116 on linux ubuntu 12.04

    Thanks for help
    -Tomas

    I have very simple HelloLambda class:

    package com.test.hello;

    public class HelloLambda {
    public static void main(String[] args) {
    process("This is Lambda", s -> System.out.println(s));
    }

    private static void process(String msg, Printer p) {
    p.out(msg);
    }

    public interface Printer {
    void out(T t);
    }
    }


    java -jar -Dretrolambda.inputDir=build/classes/main -Dretrolambda.classpath=build/classes/main -Dretrolambda.bytecodeVersion=50 -Dretrolambda.outputDir=build/libs/output -javaagent:retrolambda-1.1.0.jar retrolambda-1.1.0.jar

    and it produces exception:

    Error! Failed to transform some classes
    java.lang.RuntimeException: java.util.NoSuchElementException
    at net.orfjackal.retrolambda.LambdaReifier.reifyLambdaClass(LambdaReifier.java:32)
    at net.orfjackal.retrolambda.LambdaUsageBackporter$InvokeDynamicInsnConvertingMethodVisitor.backportLambda(LambdaUsageBackporter.java:107)
    at net.orfjackal.retrolambda.LambdaUsageBackporter$InvokeDynamicInsnConvertingMethodVisitor.visitInvokeDynamicInsn(LambdaUsageBackporter.java:99)
    at org.objectweb.asm.ClassReader.readCode(ClassReader.java:1343)
    at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:938)
    at org.objectweb.asm.ClassReader.accept(ClassReader.java:669)
    at org.objectweb.asm.ClassReader.accept(ClassReader.java:506)
    at net.orfjackal.retrolambda.LambdaUsageBackporter.transform(LambdaUsageBackporter.java:28)
    at net.orfjackal.retrolambda.Main$1.transform(Main.java:40)
    at net.orfjackal.retrolambda.BytecodeTransformingFileVisitor.visitFile(BytecodeTransformingFileVisitor.java:25)
    at net.orfjackal.retrolambda.BytecodeTransformingFileVisitor.visitFile(BytecodeTransformingFileVisitor.java:11)
    at java.nio.file.Files.walkFileTree(Files.java:2667)
    at java.nio.file.Files.walkFileTree(Files.java:2739)
    at net.orfjackal.retrolambda.Main.main(Main.java:38)
    Caused by: java.util.NoSuchElementException
    at java.util.concurrent.LinkedBlockingDeque.getFirst(LinkedBlockingDeque.java:553)
    at net.orfjackal.retrolambda.LambdaReifier.getLambdaFactoryMethod(LambdaReifier.java:47)
    at net.orfjackal.retrolambda.LambdaReifier.reifyLambdaClass(LambdaReifier.java:29)
    ... 13 more

    ReplyDelete
    Replies
    1. Esko Luontola27 November, 2013 20:32

      The "-jar" must be the last argument just before the jar file.

      Delete
    2. Esko Luontola27 November, 2013 21:39

      I've just released version 1.1.1 which gives a proper error message in this situation (it shows the usage help if the Java agent is not loaded).

      Delete
  16. Tomas Brandalik06 December, 2013 16:56

    Cool it works. I haven't noticed wrong -jar option. Thank you.

    ReplyDelete
  17. Unknown06 January, 2014 21:29

    Must be some stupid mistake, but I can't see it. I tried to reproduce the example for Dalvik mentioned above and caught a java.util.NoSuchElementException. I used the latest JDK8 from their site.

    Here's the code I used and commands I ran - http://pastebin.com/vVbSkBBh

    ReplyDelete
    Replies
    1. Esko Luontola08 January, 2014 00:34

      Thanks for reporting the problem. A recent JDK 8 had changes which made it incompatible with Retrolambda 1.1.1. I have now released Retrolambda 1.1.2 which fixes the problem.

      Delete
  18. Tomas Brandalik14 January, 2014 10:37

    Hi Esko
    haven't you tried to resolve problem of default interfaces? This is another feature that breaks our backporting from jdk 8 to 6.

    -Tomas

    ReplyDelete
    Replies
    1. Esko Luontola14 January, 2014 21:45

      No. I think that it would be hard to fully backport default interfaces, because they support polymorphism. One option might be to find all implementors of an interface that uses default methods, and copy the method implementation there, unless that class or its superclasses has an implementation. But I don't think that it would be worth the effort, because that alone would not give you the new JDK 8 APIs that were the reason for adding default methods in the first place.

      Delete
    2. Unknown14 January, 2014 22:51

      ... but still I think it's great idea :)

      And it will be useful for e.g. some API development on java 8 and re-distribute to older version. There is less code maintenance.

      I think its good approach to scan superclasses and interfaces for default method, and after collecting them all, import them in 'root' class. I'm not so much in bytecode but who knows with proper motivation and time...

      In short..you do pretty awesome yob!
      best regards

      Delete
  19. Anonymous29 March, 2014 02:06

    Nice project, but unfortunately doesnt support "invokespecial" which is my main usecase...

    so i started a sourcecode patching project https://github.com/wolfposd/to7andbeyond

    currently its really rudimentary and only changes oneliners, but already suits my needs

    ReplyDelete
    Replies
    1. Esko Luontola29 March, 2014 11:17

      I don't quite follow you that what do you mean by it not supporting invokespecial. Please give an example of Java 8 source code that doesn't work with Retrolambda.

      Delete
    2. Anonymous29 March, 2014 16:20

      myjButton.addActionlistener( e -> someMethod());

      tells me invokespecial is not supported

      Delete
    3. Esko Luontola30 March, 2014 23:52

      Now that's funny, because I have absolutely no problems doing that with with Retrolambda: https://gist.github.com/orfjackal/9879520

      Delete
    4. Anonymous31 March, 2014 17:37

      i could narrow it down to the ant task i was building, turns out the classpath-property wasn't set correctly

      Delete
  20. Stefan Zobel12 May, 2014 02:14

    Thanks for this awesome tool! It works pretty well for me.

    For what it's worth (Caution: shamleless plug): I've attempted a backport of the java.util.stream API to Java 6, using your retrolambda, that, at first sigth, appears to hold the water.

    https://sourceforge.net/projects/streamsupport/?source=directory

    If anyone is interested: It's still in early alpha, so don't expect too much yet.

    ReplyDelete
  21. Unknown14 May, 2014 13:59

    This is indeed a great news. If we can use lambda expression with its full glory in Java 6 or Java 7, then there would be lot of people who can use it, as upgrading to Java 8 is not as easy in big companies as for your fun project.

    ReplyDelete
  22. leena02 October, 2015 14:30

    Hi Esko,
    I have ran " java -jar retrolambda.jar " on command line and it worked fine.Also in my eclipse I set Jdk 1.7 for running the code and jdk 1.8 for compiling the code.But when I run the code it give me this error "Exception in thread "main" java.lang.UnsupportedClassVersionError: RunnableTest : Unsupported major.minor version 52.0"
    Do you think I have missed any step ? Thanks

    ReplyDelete
    Replies
    1. Esko Luontola03 October, 2015 23:00

      The RunnableTest class is either missing from the list of classes to be processed by Retrolambda, or something recompiles it after it was processed.

      Delete

Subscribe to: Post Comments (Atom)