Monday, April 14, 2014

RubikFX: Solving the Rubik's Cube with JavaFX 3D

Hi all!

It's really been a while since my last post... But in the middle, three major conferences kept me away from my blog. I said to myself I had to blog about the topics of my talks, but I had no chance at all. 

Now, with JavaOne still far in the distance, and after finishing my collaboration in the book, soon to be published, JavaFX 8, Introduction by Example, with my friends Carl Dea, Gerrit Grunwald, Mark Heckler and Sean Phillips, I've had the chance to play with Java 8 and JavaFX 3D for a few weeks, and this post is the result of my findings.

It happens that my kids had recently at home a Rubik's cube, and we also built the Lego Mindstorms EV3 solver thanks to this incredible project from David Gilday, the guy behind the CubeStormer 3 with the world record of fastest solving.




After playing with the cube for a while, I thought about the possibility of creating a JavaFX application for solving the cube, and that's how RubikFX was born.

If you're eager to know what this is about, here is a link to a video in YouTube which will show you most of it. 

Basically, in this post I'll talk about importing 3D models in a JavaFX scene, with the ability to zoom, rotate, scale them, add lights, move the camera,... Once we have a nice 3D model of the Rubik's cube, we will try to create a way of moving layers independently from the rest of the model, keeping track of the changes made. Finally, we'll add mouse picking for selecting faces and rotating layers.

Please read this if you're not familiar with Rubik's cube notation and the basic steps for solving it.

Before we start

By now you should know that Java 8 is GA since the 18th of March, so the code in this project is based on this version. In case you haven't done it yet, please download from here the new SDK and update your system. Also, I use NetBeans 8.0 for its support for Java 8 including lambdas and the new Streams API, among other things. You can update your IDE from here.

I use two external dependencies. One for importing the model, from a experimental project called 3DViewer, which is part of the OpenJFX project. So we need to download it and build it. The second one is from the ControlsFX project, for adding cool dialogs to the application. Download it from here.

Finally, we need a 3D model for the cube. You can build it yourself or use a free model, like this, submitted by 3dregenerator, which you can download in 3ds or OBJ formats.

Once you've got all this ingredients, it's easy to get this picture:



For that, just extract the files, rename 'Rubik's Cube.mtl' to 'Cube.mtl' and 'Rubik's Cube.obj' to 'Cube.obj', edit this file and change the third line to 'mtllib Cube.mtl', and save the file.

Now run the 3DViewer application, and drag 'Cube.obj' to the viewer. Open Settings tab, and select Lights, turn on the ambient light with a white color, and off the puntual Light 1. You can zoom in or out (mouse wheel, right button or navigation bar), rotate the cube with the left button (modifying rotation speed with Ctrl or Shift), or translate the model with both mouse buttons pressed.

Now select Options and click Wireframe, so you can see the triangle meshes used to build the model.




Each one of the 27 cubies is given a name in the obj file, like 'Block46' and so on. All of its triangles are grouped in meshes defined by the material assigned, so each cubie is made of 1 to 6 meshes, with names like 'Block46', 'Block46 (2)' and so on, and there are a total number of 117 meshes.

The color of each cubie meshes is asigned in the 'Cube.mtl' file with the Kd constant relative to the diffuse color. 

1. The Rubik's Cube - Lite Version

Importing the 3D model

So once we know how is our model, we need to construct the MeshView nodes for each mesh. The ObjImporter class from 3DViewer provide a getMeshes() method that returns a Set of names of the blocks for every mesh. So we will define a HashMap to bind every name to its MeshView. For each mesh name s we get the MeshView object with buildMeshView(s) method. 

By design, the cube materials in this model don't reflect light (Ns=0), so we'll change this to allow interaction with puntual lights, by modifying the material specular power property, defined in the PhongMaterial class.

Finally, we will rotate the original model so we have the white face on the top, and the blue one on the front.

public class Model3D {
    
    // Cube.obj contains 117 meshes, marked as "Block46",...,"Block72 (6)" in this set: 
    private Set<String> meshes;

    // HashMap to store a MeshView of each mesh with its key
    private final Map<String,Meshview> mapMeshes=new HashMap<>();
    
    public void importObj(){
        try {// cube.obj
            ObjImporter reader = new ObjImporter(getClass().getResource("Cube.obj").toExternalForm());
            meshes=reader.getMeshes(); // set with the names of 117 meshes
                      
            Affine affineIni=new Affine();            
            affineIni.prepend(new Rotate(-90, Rotate.X_AXIS));
            affineIni.prepend(new Rotate(90, Rotate.Z_AXIS));
            meshes.stream().forEach(s-> { 
                MeshView cubiePart = reader.buildMeshView(s);
                // every part of the cubie is transformed with both rotations:
                cubiePart.getTransforms().add(affineIni); 
                // since the model has Ns=0 it doesn't reflect light, so we change it to 1
                PhongMaterial material = (PhongMaterial) cubiePart.getMaterial();
                material.setSpecularPower(1);
                cubiePart.setMaterial(material);
                // finally, add the name of the part and the cubie part to the hashMap:
                mapMeshes.put(s,cubiePart); 
            });
        } catch (IOException e) {
            System.out.println("Error loading model "+e.toString());
        }
    }
    public Map<String, MeshView> getMapMeshes() {
        return mapMeshes;
    }
}

Since the model is oriented with white to the right (X axis) and red in the front (Z axis) (see picture above), two rotations are required: first rotate -90 degrees towards X axis, to put blue in the front, and then rotate 90 degrees arount Z axis to put white on top.

Mathematically, the second rotation matrix in Z must by multiplied on the left to the first matrix in X. But according to this link if we use add or append matrix rotations are operated on the right, and this will be wrong:

cubiePart.getTransforms().addAll(new Rotate(-90, Rotate.X_AXIS),
                                 new Rotate(90, Rotate.Z_AXIS));

as it will perform a first rotation in Z and a second one in X, putting red on top and yellow on front. Also this is wrong too:

cubiePart.getTransforms().addAll(new Rotate(90, Rotate.Z_AXIS),
                                  new Rotate(-90, Rotate.X_AXIS));

Though it does the right rotations, then it will require for further rotations of any cubie to be rotated from its original position, which is quite more complicated than rotating always from the last state.
           
So prepend is the right way to proceed here, and we just need to prepend the last rotation matrix to the Affine matrix of the cubie with all the previous rotations stored there.


Handling the model

After importing the obj file, we can figure out which is the number of each cubie, and once the cube it's well positioned (white face top, blue face front), the scheme we're going to use is a List<Integer> with 27 items:
  • first 9 indexes are the 9 cubies in the (F)Front face, from top left (R/W/B) to down right (Y/O/B).
  • second 9 indexes are from the (S)Standing face, from top left (R/W) to down right (Y/O).
  • last 9 indexes are from (B)Back face, from top left (G/R/W) to down right (G/Y/O).
But for performing rotations of these cubies, the best way is the internal use of a 3D array of integers:

    private final int[][][] cube={{{50,51,52},{49,54,53},{59,48,46}},
                                  {{58,55,60},{57,62,61},{47,56,63}},
                                  {{67,64,69},{66,71,70},{68,65,72}}};

where 50 is the number of the R/W/B cubie and 72 is the number for the G/Y/O.

The Rotations class will take care of any face rotation.

    // This is the method to perform any rotation on the 3D array just by swapping indexes
    // first index refers to faces F-S-B
    // second index refers to faces U-E-D
    // third index refers to faces L-M-R
    public void turn(String rot){ 
            int t = 0;
            for(int y = 2; y >= 0; --y){
                for(int x = 0; x < 3; x++){
                    switch(rot){
                        case "L":  tempCube[x][t][0] = cube[y][x][0]; break;
                        case "Li": tempCube[t][x][0] = cube[x][y][0]; break;
                        case "M":  tempCube[x][t][1] = cube[y][x][1]; break;
                        case "Mi": tempCube[t][x][1] = cube[x][y][1]; break;
                        case "R":  tempCube[t][x][2] = cube[x][y][2]; break;
                        case "Ri": tempCube[x][t][2] = cube[y][x][2]; break;
                        case "U":  tempCube[t][0][x] = cube[x][0][y]; break;
                        case "Ui": tempCube[x][0][t] = cube[y][0][x]; break;
                        case "E":  tempCube[x][1][t] = cube[y][1][x]; break;
                        case "Ei": tempCube[t][1][x] = cube[x][1][y]; break;
                        case "D":  tempCube[x][2][t] = cube[y][2][x]; break;
                        case "Di": tempCube[t][2][x] = cube[x][2][y]; break;
                        case "F":  tempCube[0][x][t] = cube[0][y][x]; break;
                        case "Fi": tempCube[0][t][x] = cube[0][x][y]; break;
                        case "S":  tempCube[1][x][t] = cube[1][y][x]; break;
                        case "Si": tempCube[1][t][x] = cube[1][x][y]; break;
                        case "B":  tempCube[2][t][x] = cube[2][x][y]; break;
                        case "Bi": tempCube[2][x][t] = cube[2][y][x]; break;
                    }
                }
                t++;
            }
        
        save();
    }

Similar rotations can be performed to the whole cube (X, Y or Z).

The content model

Once we have our model, we need a scene to display it. For that we'll use a SubScene object as content container, wrapped in a ContentModel class, where camera, lights and orientation axis are added, which is based in the ContentModel class from 3DViewer application:
 
public class ContentModel {
    public ContentModel(double paneW, double paneH, double dimModel) {
        this.paneW=paneW;
        this.paneH=paneH;
        this.dimModel=dimModel;
        buildCamera();
        buildSubScene();        
        buildAxes();
        addLights();        
    }

    private void buildCamera() {
        camera.setNearClip(1.0);
        camera.setFarClip(10000.0);
        camera.setFieldOfView(2d*dimModel/3d);
        camera.getTransforms().addAll(yUpRotate,cameraPosition,
                                      cameraLookXRotate,cameraLookZRotate);
        cameraXform.getChildren().add(cameraXform2);
        cameraXform2.getChildren().add(camera);
        cameraPosition.setZ(-2d*dimModel);
        root3D.getChildren().add(cameraXform);
        
        // Rotate camera to show isometric view X right, Y top, Z 120º left-down from each
        cameraXform.setRx(-30.0);
        cameraXform.setRy(30);

    }

    private void buildSubScene() {
        root3D.getChildren().add(autoScalingGroup);
        
        subScene = new SubScene(root3D,paneW,paneH,true,javafx.scene.SceneAntialiasing.BALANCED);
        subScene.setCamera(camera);
        subScene.setFill(Color.CADETBLUE);
        setListeners(true);
    }

    private void buildAxes() {
        double length = 2d*dimModel;
        double width = dimModel/100d;
        double radius = 2d*dimModel/100d;
        final PhongMaterial redMaterial = new PhongMaterial();
        redMaterial.setDiffuseColor(Color.DARKRED);
        redMaterial.setSpecularColor(Color.RED);
        final PhongMaterial greenMaterial = new PhongMaterial();
        greenMaterial.setDiffuseColor(Color.DARKGREEN);
        greenMaterial.setSpecularColor(Color.GREEN);
        final PhongMaterial blueMaterial = new PhongMaterial();
        blueMaterial.setDiffuseColor(Color.DARKBLUE);
        blueMaterial.setSpecularColor(Color.BLUE);
        
        Sphere xSphere = new Sphere(radius);
        Sphere ySphere = new Sphere(radius);
        Sphere zSphere = new Sphere(radius);
        xSphere.setMaterial(redMaterial);
        ySphere.setMaterial(greenMaterial);
        zSphere.setMaterial(blueMaterial);
        
        xSphere.setTranslateX(dimModel);
        ySphere.setTranslateY(dimModel);
        zSphere.setTranslateZ(dimModel);
        
        Box xAxis = new Box(length, width, width);
        Box yAxis = new Box(width, length, width);
        Box zAxis = new Box(width, width, length);
        xAxis.setMaterial(redMaterial);
        yAxis.setMaterial(greenMaterial);
        zAxis.setMaterial(blueMaterial);
        
        autoScalingGroup.getChildren().addAll(xAxis, yAxis, zAxis);
        autoScalingGroup.getChildren().addAll(xSphere, ySphere, zSphere);
    }
    
    private void addLights(){
        root3D.getChildren().add(ambientLight);
        root3D.getChildren().add(light1);
        light1.setTranslateX(dimModel*0.6);
        light1.setTranslateY(dimModel*0.6);
        light1.setTranslateZ(dimModel*0.6);
    }
}

For the camera, a Xform class from 3DViewer is used to change easily its rotation values. This also allows the initial rotation of the camera to show an isometric view:

    cameraXform.setRx(-30.0);
    cameraXform.setRy(30);

Other valid ways to perform these rotations could be based on obtaining the vector and angle of rotation to combine two rotations, which involve calculate the rotation matrix first and then the vector and angle (as I explained here):

    camera.setRotationAxis(new Point3D(-0.694747,0.694747,0.186157));
    camera.setRotate(42.1812);

Or prepending the two rotations to all the previous transformations, by appending all of them in a single Affine matrix before prepending these two rotations:

    Affine affineCamIni=new Affine();
    camera.getTransforms().stream().forEach(affineCamIni::append);
    affineCamIni.prepend(new Rotate(-30, Rotate.X_AXIS));
    affineCamIni.prepend(new Rotate(30, Rotate.Y_AXIS));
    camera.getTransforms().setAll(affineCamIni);

Then we add the listeners to the subscene, so the camera can be easily rotated.

    private void setListeners(boolean addListeners){
        if(addListeners){
            subScene.addEventHandler(MouseEvent.ANY, mouseEventHandler);
        } else {
            subScene.removeEventHandler(MouseEvent.ANY, mouseEventHandler);
        }
    }

    private final EventHandler<MouseEvent> mouseEventHandler = event -> {
        double xFlip = -1.0, yFlip=1.0; // y Up
        if (event.getEventType() == MouseEvent.MOUSE_PRESSED) {
            mousePosX = event.getSceneX();
            mousePosY = event.getSceneY();
            mouseOldX = event.getSceneX();
            mouseOldY = event.getSceneY();

        } else if (event.getEventType() == MouseEvent.MOUSE_DRAGGED) {
            double modifier = event.isControlDown()?0.1:event.isShiftDown()?3.0:1.0;

            mouseOldX = mousePosX;
            mouseOldY = mousePosY;
            mousePosX = event.getSceneX();
            mousePosY = event.getSceneY();
            mouseDeltaX = (mousePosX - mouseOldX);
            mouseDeltaY = (mousePosY - mouseOldY);

            if(event.isMiddleButtonDown() || (event.isPrimaryButtonDown() && event.isSecondaryButtonDown())) {
                cameraXform2.setTx(cameraXform2.t.getX() + xFlip*mouseDeltaX*modifierFactor*modifier*0.3); 
                cameraXform2.setTy(cameraXform2.t.getY() + yFlip*mouseDeltaY*modifierFactor*modifier*0.3);
            }
            else if(event.isPrimaryButtonDown()) {
                cameraXform.setRy(cameraXform.ry.getAngle() - yFlip*mouseDeltaX*modifierFactor*modifier*2.0);
                cameraXform.setRx(cameraXform.rx.getAngle() + xFlip*mouseDeltaY*modifierFactor*modifier*2.0);
            }
            else if(event.isSecondaryButtonDown()) {
                double z = cameraPosition.getZ();
                double newZ = z - xFlip*(mouseDeltaX+mouseDeltaY)*modifierFactor*modifier;
                cameraPosition.setZ(newZ);
            }
        }
    };

Handling the model

Now we can put all together and create the Rubik class, where the 3D model is imported, all the meshviews are created and grouped in cube, which is added to the content subscene. At the same time, rot is instantiated with the original position of the cubies.

public class Rubik {
    public Rubik(){
        // Import Rubik's Cube model and arrows
        Model3D model=new Model3D();
        model.importObj();
        mapMeshes=model.getMapMeshes();
        cube.getChildren().setAll(mapMeshes.values());
        dimCube=cube.getBoundsInParent().getWidth();
        
        // Create content subscene, add cube, set camera and lights
        content = new ContentModel(800,600,dimCube); 
        content.setContent(cube);
                
        // Initialize 3D array of indexes and a copy of original/solved position
        rot=new Rotations();
        order=rot.getCube();

        // save original position
        mapMeshes.forEach((k,v)->mapTransformsOriginal.put(k, v.getTransforms().get(0)));
        orderOriginal=order.stream().collect(Collectors.toList());
        
        // Listener to perform an animated face rotation
        rotMap=(ov,angOld,angNew)->{ 
            mapMeshes.forEach((k,v)->{
                layer.stream().filter(l->k.contains(l.toString()))
                    .findFirst().ifPresent(l->{
                        Affine a=new Affine(v.getTransforms().get(0));
                        a.prepend(new Rotate(angNew.doubleValue()-angOld.doubleValue(),axis));
                        v.getTransforms().setAll(a);
                    });
            });
        };
    }
}

Finally we create a listener for rotating layers of cubies in a Timeline animation. As the rotations are prepended to the actual affine matrix of the cubies, to perform a smooth animation we'll change the angle between 0 and 90º, and listen how the timeline internally interpolate it, making small rotations between angNew and angOld angles.

So the method to perform the rotation could be like this:

    public void rotateFace(final String btRot){
        if(onRotation.get()){
            return;
        }
        onRotation.set(true);
        
        // rotate cube indexes
        rot.turn(btRot);
        // get new indexes in terms of blocks numbers from original order
        reorder=rot.getCube();
        // select cubies to rotate: those in reorder different from order.
        AtomicInteger index = new AtomicInteger();
        layer=order.stream()
                   .filter(o->!Objects.equals(o, reorder.get(index.getAndIncrement())))
                   .collect(Collectors.toList());
        // add central cubie
        layer.add(0,reorder.get(Utils.getCenter(btRot)));
        // set rotation axis            
        axis=Utils.getAxis(btRot); 
        
        // define rotation
        double angEnd=90d*(btRot.endsWith("i")?1d:-1d);
        
        rotation.set(0d);
        // add listener to rotation changes
        rotation.addListener(rotMap);

        // create animation
        Timeline timeline=new Timeline();
        timeline.getKeyFrames().add(
            new KeyFrame(Duration.millis(600), e->{
                    // remove listener
                    rotation.removeListener(rotMap);
                    onRotation.set(false); 
                },  new KeyValue(rotation,angEnd)));
        timeline.playFromStart();

        // update order with last list
        order=reorder.stream().collect(Collectors.toList());
    }

RubikFX, Lite Version

Later on we'll add more features, but for now let's create a JavaFX application, with a BorderPane, add content to the center of the pane, and a few toolbars with buttons to perform rotations.

public class TestRubikFX extends Application {
    
    private final BorderPane pane=new BorderPane();
    private Rubik rubik;
    
    @Override
    public void start(Stage stage) {
        rubik=new Rubik();
        // create toolbars
        ToolBar tbTop=new ToolBar(new Button("U"),new Button("Ui"),new Button("F"),
                                  new Button("Fi"),new Separator(),new Button("Y"),
                                  new Button("Yi"),new Button("Z"),new Button("Zi"));
        pane.setTop(tbTop);
        ToolBar tbBottom=new ToolBar(new Button("B"),new Button("Bi"),new Button("D"),
                                     new Button("Di"),new Button("E"),new Button("Ei"));
        pane.setBottom(tbBottom);
        ToolBar tbRight=new ToolBar(new Button("R"),new Button("Ri"),new Separator(),
                                    new Button("X"),new Button("Xi"));
        tbRight.setOrientation(Orientation.VERTICAL);
        pane.setRight(tbRight);
        ToolBar tbLeft=new ToolBar(new Button("L"),new Button("Li"),new Button("M"),
                                   new Button("Mi"),new Button("S"),new Button("Si"));
        tbLeft.setOrientation(Orientation.VERTICAL);
        pane.setLeft(tbLeft);
        
        pane.setCenter(rubik.getSubScene());
        
        pane.getChildren().stream()
            .filter(n->(n instanceof ToolBar))
            .forEach(tb->{
                ((ToolBar)tb).getItems().stream()
                    .filter(n->(n instanceof Button))
                    .forEach(n->((Button)n).setOnAction(e->rubik.rotateFace(((Button)n).getText())));
            });
        rubik.isOnRotation().addListener((ov,b,b1)->{
            pane.getChildren().stream()
            .filter(n->(n instanceof ToolBar))
            .forEach(tb->tb.setDisable(b1));
        });
        final Scene scene = new Scene(pane, 880, 680, true);
        scene.setFill(Color.ALICEBLUE);
        stage.setTitle("Rubik's Cube - JavaFX3D");
        stage.setScene(scene);
        stage.show();
    } 
}

Now this is what we have already accomplished:


If you're interested in having a deeper look at the application, you can find the source code in my GitHub repository. Note you'll need to add the 3DViewer.jar. 

How does it work?

Take, for instance, a initial "F" rotation. We apply it to rot:


        // rotate cube indexes
        rot.turn(btRot);
        // get new indexes in terms of blocks numbers from original order
        reorder=rot.getCube();

Using rot.printCube() we can see the numbers of cubies for the solved cube (order) and for the new one, with the frontal layer rotated clockwise (reorder):

order:    50 51 52 49 54 53 59 48 46 || 58 55 60 57 62 61 47 56 63 || 67 64 69 66 71 70 68 65 72
reorder:  59 49 50 48 54 51 46 53 52 || 58 55 60 57 62 61 47 56 63 || 67 64 69 66 71 70 68 65 72


By comparing both lists and getting the different items, we know which cubies must be rotated, though we have to add the number of the central cubie (54), as it is the same in both lists, but it should be rotated too. So we create the list layer with these nine cubies:

    // select cubies to rotate: those in reorder different from order.
        AtomicInteger index = new AtomicInteger();
        layer=order.stream()
                   .filter(o->!Objects.equals(o, reorder.get(index.getAndIncrement())))
                   .collect(Collectors.toList());
        // add central cubie
        layer.add(0,reorder.get(Utils.getCenter(btRot)));
        // set rotation axis            
        axis=Utils.getAxis(btRot); 

Utils is a class that manage the values for each type of rotation. For this case:

   public static Point3D getAxis(String face){
        Point3D p=new Point3D(0,0,0);
        switch(face.substring(0,1)){
            case "F":  
            case "S":  p=new Point3D(0,0,1); 
                       break;
        }
        return p;
    }
    
    public static int getCenter(String face){
        int c=0;
        switch(face.substring(0,1)){
            case "F":  c=4;  break;
        }
        return c;
    }

Once we've got the cubies and the axis of rotation, now it's worth noticing how the rotation listener works. With the timeline, the angle goes from 0 to 90º with an EASE_BOTH interpolation (by default), so the angle increments are smaller at the beginning, bigger in the middle and smaller again at the end. This could be a possible list of increments: 0.125º-3º-4.6º-2.2º-2.48º-...-2.43º-4.78º-2.4º-2.4º-0.55º.

For every value in angNew, the listener rotMap applies a small rotation to a layer of cubies. For that we look in our HashMap which meshviews belongs to these cubies, and prepend a new rotation to their previous affine matrix:

   // Listener to perform an animated face rotation
        rotMap=(ov,angOld,angNew)->{ 
            mapMeshes.forEach((k,v)->{
                layer.stream().filter(l->k.contains(l.toString()))
                    .findFirst().ifPresent(l->{
                        Affine a=new Affine(v.getTransforms().get(0));
                        a.prepend(new Rotate(angNew.doubleValue()-angOld.doubleValue(),axis));
                        v.getTransforms().setAll(a);
                    });
            });
        };

So in 600 ms we apply around 30 to 40 small rotations to a bunch of around 40 meshviews. 

Finally, after the rotation is done, we just need to update order with the last list of cubies, so we can start all over again with a new rotation.


2. The Rubik's Cube - Full Version

 Adding more features

Now that we've got a working but pretty basic Rubik's cube JavaFX application, it's time for adding a few extra features, like graphic arrows and preview rotations to show the direction of rotation before they're performed.

Scramble and Sequences

Let's start by adding a scramble routine, to scramble the cubies before start solving the cube. To do that we generate a sequence of 25 random moves from a list of valid rotations.

   private static final List<String> movements = 
    Arrays.asList("F", "Fi", "F2", "R", "Ri", "R2", 
                  "B", "Bi", "B2", "L", "Li", "L2",
                  "U", "Ui", "U2", "D", "Di", "D2");

    private String last="V", get="V";
    public void doScramble(){
        StringBuilder sb=new StringBuilder();
        IntStream.range(0, 25).boxed().forEach(i->{
            while(last.substring(0, 1).equals(get.substring(0, 1))){
                // avoid repeating the same/opposite rotations
                get=movements.get((int)(Math.floor(Math.random()*movements.size())));
            }
            last=get;
            if(get.contains("2")){
                get=get.substring(0,1);
                sb.append(get).append(" ");
            }
            sb.append(get).append(" ");
        });
        doSequence(sb.toString().trim());
    }

Then we have to perform this sequence, by rotating each movement.  First we extract the rotations from the string, converting other notations (like lower letters or ' instead of 'i' for counter clockwise rotations) to the one used.

A listener is added to onRotation, so only when the last rotation finishes, a new rotation starts. By adding a second listener to the index property, when the end of the list plus one is reached, this listener is stopped, allowing for the last rotation to finish properly, and saving the rotations for a further replay option.

   public void doSequence(String list){
        onScrambling.set(true);
        List<String> asList = Arrays.asList(list.replaceAll("’", "i").replaceAll("'", "i").split(" "));
        
        sequence=new ArrayList<>();
        asList.stream().forEach(s->{
            if(s.contains("2")){
                sequence.add(s.substring(0, 1));
                sequence.add(s.substring(0, 1));            
            } else if(s.length()==1 && s.matches("[a-z]")){
                sequence.add(s.toUpperCase().concat("i"));
            } else {
                sequence.add(s);
            }
        });
        System.out.println("seq: "+sequence);
        
        IntegerProperty index=new SimpleIntegerProperty(1);
        ChangeListener<boolean> lis=(ov,b,b1)->{
            if(!b1){
                if(index.get()<sequence.size()){
                    rotateFace(sequence.get(index.get()));
                } else {
                    // save transforms
                    mapMeshes.forEach((k,v)->mapTransformsScramble.put(k, v.getTransforms().get(0)));
                    orderScramble=reorder.stream().collect(Collectors.toList());
                } 
                index.set(index.get()+1);
            }
        };
        index.addListener((ov,v,v1)->{
            if(v1.intValue()==sequence.size()+1){
                onScrambling.set(false);
                onRotation.removeListener(lis);
                count.set(-1);
            }
        });
        onRotation.addListener(lis);
        rotateFace(sequence.get(0));
    }

Note that we use a Dialog from ControlsFX to prevent losing previous moves.


   Button bSc=new Button("Scramble");
        bSc.setOnAction(e->{
            if(moves.getNumMoves()>0){
                Action response = Dialogs.create()
                .owner(stage)
                .title("Warning Dialog")
                .masthead("Scramble Cube")
                .message( "You will lose all your previous movements. Do you want to continue?")
                .showConfirm();
                if(response==Dialog.Actions.YES){
                    rubik.doReset();
                    doScramble();
                }
            } else {
                doScramble();
            }
        });

If you want to load a sequence, like any of these, another Dialog with input allowed is used.

   Button bSeq=new Button("Sequence");
        bSeq.setOnAction(e->{
            String response;
            if(moves.getNumMoves()>0){
                response = Dialogs.create()
                .owner(stage)
                .title("Warning Dialog")
                .masthead("Loading a Sequence").lightweight()
                .message("Add a valid sequence of movements:\n(previous movements will be discarded)")
                .showTextInput(moves.getSequence());
            } else {
                response = Dialogs.create()
                .owner(stage)
                .title("Information Dialog")
                .masthead("Loading a Sequence").lightweight()
                .message( "Add a valid sequence of movements")
                .showTextInput();
            }
            if(response!=null && !response.isEmpty()){
                rubik.doReset();
                rubik.doSequence(response.trim());
            }
        });

The results of scrambling a cube or adding a sequence of rotations can be seen in this video.



Timer and moves counter

Let's add now a timer using the new Date and Time API for Java 8. You may have noticed the timer in the bottom toolbar in the previous video.

For that, we use the following code in RubikFX class:


    private LocalTime time=LocalTime.now();
    private Timeline timer;
    private final StringProperty clock = new SimpleStringProperty("00:00:00");
    private final DateTimeFormatter fmt = DateTimeFormatter.ofPattern("HH:mm:ss").withZone(ZoneId.systemDefault());
    
    @Override
    public void start(Stage stage) {
        ...
        Label lTime=new Label();
        lTime.textProperty().bind(clock);
        tbBottom.getItems().addAll(new Separator(),lTime);

        timer=new Timeline(new KeyFrame(Duration.ZERO, e->{
            clock.set(LocalTime.now().minusNanos(time.toNanoOfDay()).format(fmt));
        }),new KeyFrame(Duration.seconds(1)));
        timer.setCycleCount(Animation.INDEFINITE);

        rubik.isSolved().addListener((ov,b,b1)->{
            if(b1){
                timer.stop();
            }
        });
        
        time=LocalTime.now();
        timer.playFromStart();
    }

For the counter, we'll add two classes. Move is a simple POJO class, with a string for the name of the rotation and a long for the timestamp of the movement. Moves class will contain a list of moves.


public class Moves {
    private final List<Move> moves=new ArrayList<>();
    
    public Moves(){
        moves.clear();
    }
    
    public void addMove(Move m){ moves.add(m); }
    public List<Move> getMoves() { return moves; }
    public Move getMove(int index){
        if(index>-1 && index<moves.size()){
            return moves.get(index);
        }
        return null;
    }
    public String getSequence(){
        StringBuilder sb=new StringBuilder("");
        moves.forEach(m->sb.append(m.getFace()).append(" "));
        return sb.toString().trim();
    }
}

For adding the number of rotations, we use the following code in RubikFX class:


    private Moves moves=new Moves();
    
    @Override
    public void start(Stage stage) {
        ...
        rubik.getLastRotation().addListener((ov,v,v1)->{
            if(!v1.isEmpty()){
                moves.addMove(new Move(v1, LocalTime.now().minusNanos(time.toNanoOfDay()).toNanoOfDay()));
            }
        });

        Label lMov=new Label();
        rubik.getCount().addListener((ov,v,v1)->{
            lMov.setText("Movements: "+(v1.intValue()+1));
        });
        tbBottom.getItems().addAll(new Separator(),lMov);
    }

Replay

We can also replay the list of movements the user has done stored in moves. For that we need to restore first the state of the cube right after the scramble, and performe one by one all the rotations from the list.


    public void doReplay(List<Move> moves){
        if(moves.isEmpty()){
            return;
        }
        content.resetCam();
        //restore scramble
        if(mapTransformsScramble.size()>0){
            mapMeshes.forEach((k,v)->v.getTransforms().setAll(mapTransformsScramble.get(k)));
            order=orderScramble.stream().collect(Collectors.toList());
            rot.setCube(order);
            count.set(-1);
        } else {
            // restore original
            doReset();
        }

        onReplaying.set(true);        
        IntegerProperty index=new SimpleIntegerProperty(1);
        ChangeListener lis=(ov,v,v1)->{
            if(!v1 && moves.size()>1){
                if(index.get()<moves.size()){
                    timestamp.set(moves.get(index.get()).getTimestamp());
                    rotateFace(moves.get(index.get()).getFace());
                }
                index.set(index.get()+1);
            }
        };
        index.addListener((ov,v,v1)->{
            if(v1.intValue()==moves.size()+1){
                onReplaying.set(false);
                onRotation.removeListener(lis);
                acuAngle=0;
            }
        });
        onRotation.addListener(lis);
        timestamp.set(moves.get(0).getTimestamp());
        rotateFace(moves.get(0).getFace());
    }

Rotation direction preview

Time for a new feature: 3D arrows will be shown in the rotating face or axis, to show the direction.

Actually, JavaFX 3D API doesn't supply any way of building 3D complex models. There's an impressive ongoing work by Michael Hoffer to provide a way by using Constructive Solid Geometry (CSG) here, kudos Michael!!

By using primitives and boolean operations with CSG you can build a model, and even export it with STL format.




You can use free or commercial 3D software for this task too. I designed these arrows with SketchUp Make and exported them to OBJ format so I could import them as we did with the cube using ObjImporter from 3DViewer. 



While the design is fast, it requires manual editting of the created obj file to convert long faces of more than 4 vertixes as they are not properly imported.



Other approach could be exporting the file to *.3ds and use the proper importer from August Lammersdorf.  


Edit: Michael Hoffer kindly added an option to export to OBJ format, so now it would be possible to import the arrow model generated with CSG in JavaFXScad in our scene. Thanks Michael!

Once we have the model, we have to add it, scale and rotate it, so we can show the arrow in the rotating face.




For a rotation like 'Ui':
 
    public void updateArrow(String face, boolean hover){
        boolean bFaceArrow=!(face.startsWith("X")||face.startsWith("Y")||face.startsWith("Z"));
        MeshView arrow=bFaceArrow?faceArrow:axisArrow;
        
        if(hover && onRotation.get()){
            return;
        }
        arrow.getTransforms().clear();    
        if(hover){
            double d0=arrow.getBoundsInParent().getHeight()/2d;
            Affine aff=Utils.getAffine(dimCube, d0, bFaceArrow, face);
            arrow.getTransforms().setAll(aff);
            arrow.setMaterial(Utils.getMaterial(face));
            if(previewFace.get().isEmpty()) {
                previewFace.set(face);
                onPreview.set(true);
                rotateFace(face,true,false);
            }
        } else if(previewFace.get().equals(face)){
            rotateFace(Utils.reverseRotation(face),true,true);
        } else if(previewFace.get().equals("V")){
            previewFace.set("");
            onPreview.set(false);
        }
    }

where the affine is calculated in the Utils class for the current face:

    public static Affine getAffine(double dimCube, double d0, boolean bFaceArrow, String face){
        Affine aff=new Affine(new Scale(3,3,3));
        aff.append(new Translate(0,-d0,0));
        switch(face){
            case "U":   
            case "Ui":  aff.prepend(new Rotate(face.equals("Ui")?180:0,Rotate.Z_AXIS));
                        aff.prepend(new Rotate(face.equals("Ui")?45:-45,Rotate.Y_AXIS));
                        aff.prepend(new Translate(0,dimCube/2d,0));
                        break;
        }
        return aff;
    }

To trigger the drawing of the arrow, we set a listener to the buttons on the toolbars based on the mouse hovering.

We can also add a small rotation (5º) as preview of the full rotation (90º) in the face selected, by calling rotateFace again, with bPreview=true at this point.



If the user clicks on the button, the rotation is completed (from 5º to 90º). Otherwise the rotation is cancelled (from 5º to 0º). In both cases, with a smooth animation.



Select rotation by picking

Finally, the rotation could be performed based on the mouse picking of a cubie face, with visual aid showing the arrow and performing a small rotation of 5º. If the mouse is dragged far enough the full rotation will be performed after it is released. If the mouse is released close to the origin, the rotation is cancelled.

For this feature, the critical part is being able to know which mesh we are selecting with the mouse click. And for that, the API provides MouseEvent.getPickResult().getIntersectedNode(), which returns one of the meshviews on the cube.

So the next step is find which is this meshview and what cubie does it belongs to. As all the meshes have a name, like 'Block46 (2)', looking at the number of block we identify the cubie.

Now we need to find which of the faces we have selected. For that we use the triangles coordinates of the mesh, as for the faces they define a plane, so with the cross product we know the normal direction of this plane. Note we must update the operations with the actual set of transformations applied.

    private static Point3D getMeshNormal(MeshView mesh){
        TriangleMesh tm=(TriangleMesh)mesh.getMesh();
        float[] fPoints=new float[tm.getPoints().size()];
        tm.getPoints().toArray(fPoints);
        Point3D BA=new Point3D(fPoints[3]-fPoints[0],fPoints[4]-fPoints[1],fPoints[5]-fPoints[2]);
        Point3D CA=new Point3D(fPoints[6]-fPoints[0],fPoints[7]-fPoints[1],fPoints[8]-fPoints[2]);
        Point3D normal=BA.crossProduct(CA);
        Affine a=new Affine(mesh.getTransforms().get(0));
        return a.transform(normal.normalize());
    }

    public static String getPickedRotation(int cubie, MeshView mesh){
        Point3D normal=getMeshNormal(mesh);
        String rots=""; // Rx-Ry 
        switch(cubie){
            case 0: rots=(normal.getZ()>0.99)?"Ui-Li":
                            ((normal.getX()<-0.99)?"Ui-F":((normal.getY()>0.99)?"Ui-Li":""));
                    break;
        }
        return rots;
    }

Once we have the normal, we can provide the user with two possible rotations (and their possible two directions). To select which one to perform, we'll look how the user moves the mouse while it's being dragged. Note the mouse coordinates are 2D.

    public static String getRightRotation(Point3D p, String selFaces){
        double radius=p.magnitude();
        double angle=Math.atan2(p.getY(),p.getX());
        String face="";
        if(radius>=radMinimum && selFaces.contains("-") && selFaces.split("-").length==2){
            String[] faces=selFaces.split("-");
            // select rotation if p.getX>p.getY
            if(-Math.PI/4d<=angle && angle<Math.PI/4d){ // X
                face=faces[0];
            } else if(Math.PI/4d<=angle && angle<3d*Math.PI/4d){ // Y
                face=faces[1];
            } else if((3d*Math.PI/4d<=angle && angle<=Math.PI) || 
                      (-Math.PI<=angle && angle<-3d*Math.PI/4d)){ // -X
                face=reverseRotation(faces[0]);
            } else { //-Y
                face=reverseRotation(faces[1]);
            }
            System.out.println("face: "+face);
        } else if(!face.isEmpty() && radius<radMinimum){ // reset previous face
            face="";
        }
        return face;
    }

Now that we have the layer to rotate, we make a small rotation as a preview of rotation if the mouse is dragged far from the initial click point, with a minimum distance. Then if the user releases the mouse and the distance from the initial point is greater than a distance radClick, the rotation is completed. But if the distance is lower or the mouse is dragged under the distance radMinimum, the rotation is cancelled.

The next listing shows an EventHandler<MouseEvent> implemented to provide this behaviour. Note that we have to stop the camera rotations while we are picking a face and rotating a layer.

    public EventHandler<MouseEvent> eventHandler=(MouseEvent event)->{
            if (event.getEventType() == MouseEvent.MOUSE_PRESSED ||
                event.getEventType() == MouseEvent.MOUSE_DRAGGED || 
                event.getEventType() == MouseEvent.MOUSE_RELEASED) {
                
                mouseNewX = event.getSceneX();
                mouseNewY = -event.getSceneY();
                
                if (event.getEventType() == MouseEvent.MOUSE_PRESSED) {
                    Node picked = event.getPickResult().getIntersectedNode();
                    if(null != picked && picked instanceof MeshView) {
                        mouse.set(MOUSE_PRESSED);
                        cursor.set(Cursor.CLOSED_HAND);
                        stopEventHandling();
                        stopEvents=true;
                        pickedMesh=(MeshView)picked;
                        String block=pickedMesh.getId().substring(5,7);
                        int indexOf = order.indexOf(new Integer(block));
                        selFaces=Utils.getPickedRotation(indexOf, pickedMesh);
                        mouseIniX=mouseNewX;
                        mouseIniY=mouseNewY;
                        myFace=""; 
                        myFaceOld="";
                    }
                } else if (event.getEventType() == MouseEvent.MOUSE_DRAGGED) {
                    if(stopEvents && !selFaces.isEmpty()){
                        mouse.set(MOUSE_DRAGGED);
                        Point3D p=new Point3D(mouseNewX-mouseIniX,mouseNewY-mouseIniY,0);
                        radius=p.magnitude();

                        if(myFaceOld.isEmpty()){
                            myFace=Utils.getRightRotation(p,selFaces);
                            if(!myFace.isEmpty() && !onRotation.get()){
                                updateArrow(myFace, true);
                                myFaceOld=myFace;
                            } 
                            if(myFace.isEmpty()){
                                myFaceOld="";
                            }
                        }
                        // to cancel preselection, just go back to initial click point
                        if(!myFaceOld.isEmpty() && radius<Utils.radMinimum){
                            myFaceOld="";
                            updateArrow(myFace, false);
                            myFace="";
                        }
                    }
                } else if (stopEvents && event.getEventType() == MouseEvent.MOUSE_RELEASED) {
                    mouse.set(MOUSE_RELEASED);
                    if(!onRotation.get() && !myFace.isEmpty() && !myFaceOld.isEmpty()){
                        if(Utils.radClick<radius){
                            // if hand is moved far away do full rotation
                            rotateFace(myFace);
                        } else { 
                            // else preview cancellation
                            updateArrow(myFace, false);
                        }
                    }
                    myFace=""; myFaceOld="";
                    stopEvents=false;
                    resumeEventHandling();                        
                    cursor.set(Cursor.DEFAULT);
                }
            }
        };

Finally, we add this EventHandler to the scene:

scene.addEventHandler(MouseEvent.ANY, rubik.eventHandler);

This video shows how this event handling works. 


Check if cube is solved

Finally, let's add a check routine. We know the initial solved order of cubies, but we need to take into account any of the 24 possible orientations of the faces, which can be acchieved with up to two rotations.

private static final List<String> orientations=Arrays.asList("V-V","V-Y","V-Yi","V-Y2",
                                                    "X-V","X-Z","X-Zi","X-Z2",
                                                    "Xi-V","Xi-Z","Xi-Zi",
                                                    "X2-V","X2-Z","X2-Zi",
                                                    "X-Y","X-Yi","X-Y2",
                                                    "Xi-Y","Xi-Yi","X2-Y","X2-Yi",
                                                    "Z-V","Zi-V","Z2-V");
    
    public static boolean checkOrientation(String r, List<Integer> order){
        Rotations rot=new Rotations();
        for(String s:r.split("-")){
            if(s.contains("2")){
                rot.turn(s.substring(0,1));
                rot.turn(s.substring(0,1));
            } else {
                rot.turn(s);
            }
        }
        return order.equals(rot.getCube());
    }

So after any movement we have to check if the actual order matches any of these 24 solutions. For that we can use parallelStream() with a filter in which we rotate a new cube to one of the possible orientations and check if that matches the actual one:

    public static boolean checkSolution(List<Integer> order) {
        return Utils.getOrientations().parallelStream()
                .filter(r->Utils.checkOrientation(r,order)).findAny().isPresent();
    }
    
Conclusions

All along this post, we've been discussing the new JavaFX 3D API. Powerfull enough, but with lack of some usual tools in the 3D modelling world. Maybe they will come soon...

We've used the new lambdas and Stream API. I hope by now you've got a clear view of what you can do with them. For sure, they will definetely change the way we write code.

The Rubik's cube application has proven to be a nice way of testing these new capabilities, while enjoying playing, humm, I mean, developing the code.

This final video shows most of what we've accomplished in this post. As I said, it's for begginers like me with the Rubik's cube...


In my repo you can find all the code for this full version. Fill free to fork it and play with it. There're tons of improvements to make, so any pull request will be welcome!

As always, thanks for reading me! Please, try it for yourself and share any comment you may have.


Sunday, June 9, 2013

Leap Motion Controller and JavaFX: A new touch-less approach

Hi, it's been a while since my last post, but this first half of the year I've been quite busy at work. So I had to put on hold most of my JavaFX, Raspberry Pi and Arduino projects.

In all this time I could (even) afford only one distraction, because the device really deserve it!

In April I had the chance to get involved in the Leap Motion Developer program (thanks for that to Jim Weaver and Simon Ritter, and of course, to the Leap Motion staff), and since I received the little but powerful device at home, I've been playing around with it in several JavaFX based projects. 

So this post is a little briefing of the few projects I've done with the Leap Motion controller and JavaFX, most of them just as a Proof of Concept.

As the Leap SDK is private for now (though they intend to make it public soon), I won't release any code, just snippets, few screenshots and short videos.

At a glance, this is what I'll cover:
  • The Leap Motion Controller, what you can expect: hands, fingers, tools and basic gestures.
  • The JavaFX and the Leap threads, change listeners to the rescue.
  • POC #1. Moving 2D shapes on a JavaFX scene, trapping basic gestures with the Leap.
  • POC #2. Physics 2D worlds and Leap, a JavaFX approach.
  • POC #3. JavaFX 3D and Leap, with JDK8 early preview and openJFX Project.
I won't go in much detail regarding the Leap itself. There're plenty of videos out there. If you don't know about it yet, check these, for instance:
For those of you already on the pre-order list, the 22nd of July it's already there... be (just a little bit more) patient! For those who haven't decided to buy one yet, maybe this reading will help you make up your mind.

Let's go!

1. The Leap Motion Controller

After you plug in the Leap Motion device in your USB port (Windows, Mac and Linux OS), and download and install its sofware, you can try the Leap Visualizer, a bundled application which allows you to learn and discover the magic of the Leap.



As soon as you launch it, you can virtually see your hands and fingers moving around the screen. It's really impressive because of the high precision of the movements due to the high frequency in which the Leap scans.

Activating hands and fingers visualization you can realize those are the basics of the model provided: the Leap will detect none, one or several hands, and several fingers in each one of them. For each hand, it will show, for instance, its position, where it points at (hand direction) and its palm normal. For fingers, you'll get their position and where they point at. Also you can get hand or fingers velocity.

These positions, directions and velocities of your real hands and fingers are 3D vectors, refered to a right-handed Cartesian coordinate system, with the origin at the center of the device and the X and Z axes lying in the horizontal plane and the Y axis is vertical.

It's important to notice you'll have to convert these coordinates to the ones of the screen if you want to display and move anything on it. For that, you need to calculate where the vector of the hand or finger direction intersects with the plane of the screen. 

The Leap device performs complete scans of its surroundings, with an effective range approximately from 25 to 600 millimeters above the device (1 inch to 2 feet). Each scan defines a frame, with all the data associated.

The scan rate is really high, that's what makes the Leap so impressive and accurate compared to similar devices. Depending on your CPU and the amount of data analyzed, the range of processing latency goes from 2 ms to 33 ms, giving rates from 30 to 500 fps.


Besides directions, a basic collection of gestures is also provided: key tap or screen tap, swipe and circle gestures are tracked by comparing the finger movements through different frames.


The good thing of having access to all frames data is that you can define your custom gestures, and try to find them analyzing a relative short collection of frames, all over again.

To end this brief intro to the great Leap Motion device, let's say that being on the Developer Program you can get your SDK for many programming languages, such as Java, JavaScript, C++, C#, Objetive C or Phyton. 

In terms of Java code, all you need to do is extend the Listener class provided by the SDK and basically override the onFrame method, and let the magic begin.

2. The JavaFX and the Leap threads
 
Having a Leap Motion Controller means you can interact with your applications in a very different way you're used to. For that, you just need to integrate the Leap events, in terms of movement or actions, in your apps.

In a JavaFX based application, one easy way to do this is by adding ObjectProperty<T> objects to the LeapListener class in order to set desired values at every frame using Vector, Point2D, Point3D, CircleGesture,... and then implement their related public ObservableValue<T> methods. 

Then, in the JavaFX thread, an anonimous ChangeListener<T> class can be added to listen for any change in the ObservableValue. Special care must be taken here, as anything related to the UI must be deal by Platform.runLater().

The next proof of concept samples will try to explain this.

3. POC #1. Moving 2D shapes on a JavaFX scene

Let's say we want to move a node in the scene with our hand as a first simple POC.

We create two classes: SimpleLeapListener class, that extends Listener, where we just set at every frame the coordinates of the screen where the hand points at:

public class SimpleLeapListener extends Listener {

    private ObjectProperty<Point2D> point=new SimpleObjectProperty<>();
    
    public ObservableValue<Point2D> pointProperty(){ return point; }
    
    @Override
    public void onFrame(Controller controller) {
        Frame frame = controller.frame();
        if (!frame.hands().empty()) {
            Screen screen = controller.calibratedScreens().get(0);
            if (screen != null && screen.isValid()){
                Hand hand = frame.hands().get(0);
                if(hand.isValid()){
                    Vector intersect = screen.intersect(hand.palmPosition(),hand.direction(), true);
                    point.setValue(new Point2D(screen.widthPixels()*Math.min(1d,Math.max(0d,intersect.getX())),
                            screen.heightPixels()*Math.min(1d,Math.max(0d,(1d-intersect.getY())))));
                }
            }
        }
    }
}

And LeapJavaFX, our JavaFX class, that listen to changes in this point and reflect them on the scene:

 
public class LeapJavaFX extends Application { 
    private SimpleLeapListener listener = new SimpleLeapListener();
    private Controller leapController = new Controller();
    
    private AnchorPane root = new AnchorPane();
    private Circle circle=new Circle(50,Color.DEEPSKYBLUE);
    
    @Override
    public void start(Stage primaryStage) {
        
        leapController.addListener(listener);        
        circle.setLayoutX(circle.getRadius());
        circle.setLayoutY(circle.getRadius());
        root.getChildren().add(circle);
        final Scene scene = new Scene(root, 800, 600);        
        
        listener.pointProperty().addListener(new ChangeListener<point2d>(){
            @Override 
            public void changed(ObservableValue ov, Point2D t, final Point2D t1) {
                Platform.runLater(new Runnable(){
                    @Override 
                    public void run() {
                        Point2D d=root.sceneToLocal(t1.getX()-scene.getX()-scene.getWindow().getX(),
                                                    t1.getY()-scene.getY()-scene.getWindow().getY());
                        double dx=d.getX(), dy=d.getY();
                        if(dx>=0d && dx<=root.getWidth()-2d*circle.getRadius() && 
                           dy>=0d && dy<=root.getHeight()-2d*circle.getRadius()){
                            circle.setTranslateX(dx);
                            circle.setTranslateY(dy);                                
                        }
                    }
                });
            }
        });
        
        primaryStage.setScene(scene);
        primaryStage.show();
    }
    @Override
    public void stop(){
        leapController.removeListener(listener);
        
    }
}

Pretty simple, isn't it? This short video shows the result.


Here goes a second sample, based on the same idea, one circle per detected hand it's displayed, with its radius growing or shrinking according the Z distance of the hand to the Leap. When key tap kind of gestures are detected, a shadow circle is shown where the tap occurs, moved from the previous tap location with an animation of the changes in both transition and scale properties. 

Here you can see it in action:
 


4. POC #2. Physics 2D worlds and Leap, a JavaFX approach

When Toni Epple saw this video, he suggested me to add some physics to the mix, so I started learning from his blog posts about JavaFX and JBox2D, the Java port of the popular Box2D physics engine. Using his amazing work I was able to create a simple World, add some dynamic bodies and static walls to the boundaries, and a static big circle which I could move with the Leap, as in the previous samples. Thank you, Toni, your work is really inspiring!

Here is a code snippet of the JavaFX class.

public class PhysicsLeapJavaFX extends Application { 
    private SimpleLeapListener listener = new SimpleLeapListener();
    private Controller leapController = new Controller();
    
    private Button button=new Button("Add Ball");
    private AnchorPane root = new AnchorPane();
    private AnchorPane pane = new AnchorPane();
    private Body myCircle=null;
    
    private World world=null;
    private WorldView worldView=null;
    private final float worldScale=50f;
    private final float originX=4f, originY=8f;
    private final float radius=1f;

    @Override
    public void start(Stage primaryStage) {
        
        leapController.addListener(listener); 
        
        world = new World(new Vec2(0, 0f)); // No gravity
        // 200x400 -> world origin->(4f, 8f), Y axis>0 UP        
        worldView=new WorldView(world, originX*worldScale, originY*worldScale, worldScale);
        
        AnchorPane.setBottomAnchor(pane, 20d); AnchorPane.setTopAnchor(pane, 50d);
        AnchorPane.setLeftAnchor(pane, 20d);   AnchorPane.setRightAnchor(pane, 20d);
        // root: 800x600, pane: 760x530, worldScale= 50 -> world dimensions: 15.2f x 10.6f 
        pane.getChildren().setAll(worldView);        
                
        NodeManager.addProvider(new MyNodeProvider());
        
        button.setLayoutX(30); button.setLayoutY(15);
        button.setOnAction(new EventHandler<ActionEvent>(){
            @Override
            public void handle(ActionEvent t) {
                Body ball=new CircleShapeBuilder(world).userData("ball")
                            .position(0f, 4f)
                            .type(BodyType.DYNAMIC).restitution(1f).density(0.4f)
                            .radius(0.5f).friction(0f)
                            .build();
                ball.setLinearVelocity(new Vec2(4,2));
                ball.setLinearDamping(0f);
            }            
        });
        
        myCircle=new CircleShapeBuilder(world).userData("hand1").position(0f, 2f)
                .type(BodyType.STATIC).restitution(1f).density(1)
                .radius(radius).friction(0f)
                .build();
        new BoxBuilder(world).position(3.6f, 8f).restitution(1f).friction(0f)
                              .halfHeight(0.1f).halfWidth(7.7f).build();
        new BoxBuilder(world).position(3.6f, -2.6f).restitution(1f).friction(0f)
                              .halfHeight(0.1f).halfWidth(7.7f).build();
        new BoxBuilder(world).position(-4f, 2.7f).restitution(1f).friction(0f)
                              .halfHeight(5.4f).halfWidth(0.1f).build();
        new BoxBuilder(world).position(11.2f, 2.7f).restitution(1f).friction(0f)
                              .halfHeight(5.4f).halfWidth(0.1f).build();
        
        root.getChildren().addAll(button, pane);
        final Scene scene = new Scene(root, 800, 600);        
        
        listener.pointProperty().addListener(new ChangeListener<point2D>(){
            @Override 
            public void changed(ObservableValue<? extends Point2D> ov, Point2D t, final Point2D t1) {
                Platform.runLater(new Runnable(){
                    @Override 
                    public void run() {
                        Point2D d=pane.sceneToLocal(t1.getX()-scene.getX()-scene.getWindow().getX()-root.getLayoutX(),
                                                    t1.getY()-scene.getY()-scene.getWindow().getY()-root.getLayoutY());
                        double dx=d.getX()/worldScale, dy=d.getY()/worldScale;
                        if(dx>=0.1 && dx<=pane.getWidth()/worldScale-2d*radius-0.1 && 
                           dy>=0.1 && dy<=pane.getHeight()/worldScale-2d*radius-0.1){
                            myCircle.setTransform(new Vec2((float)(dx)-(originX-radius),
                                                           (originY-radius)-(float)(dy)),
                                                  myCircle.getAngle());
                        }
                    }
                });
            }
        });
        listener.keyTapProperty().addListener(new ChangeListener<Boolean>(){
            @Override public void changed(ObservableValue<? extends Boolean> ov, Boolean t, final Boolean t1) {
                if(t1.booleanValue()){
                    Platform.runLater(new Runnable(){
                        @Override public void run() {
                            button.fire();
                        }
                    });
                }
            }
        });

        primaryStage.setTitle("PhysicsLeapJavaFX Sample");
        primaryStage.setScene(scene);
        primaryStage.show();
    }
    @Override
    public void stop(){
        leapController.removeListener(listener);
        
    }
}

I've added some gesture recognition to fire the button when a key tap gesture it's done. Besides, it's quite convinient to smooth the readings from the Leap, taking the average of the last positions instead of the position for every frame. So let's modify the SimpleLeapListener class adding a size limited LinkedList collection to store the last 30 positions, and also enable the key tap gestures:

public class SimpleLeapListener extends Listener {

    private ObjectProperty<Point2D> point=new SimpleObjectProperty<>();
    public ObservableValue<Point2D> pointProperty(){ return point; }
    private LimitQueue<Vector> positionAverage = new LimitQueue<Vector>(30);
    
    private BooleanProperty keyTap= new SimpleBooleanProperty(false);
    public BooleanProperty keyTapProperty() { return keyTap; }

    @Override
    public void onFrame(Controller controller) {
        Frame frame = controller.frame();
        if (!frame.hands().empty()) {
            Screen screen = controller.calibratedScreens().get(0);
            if (screen != null && screen.isValid()){
                Hand hand = frame.hands().get(0);
                if(hand.isValid()){
                    Vector intersect = screen.intersect(hand.palmPosition(),hand.direction(), true);
                    positionAverage.add(intersect);
                    Vector avIntersect=Average(positionAverage);
                    point.setValue(new Point2D(screen.widthPixels()*Math.min(1d,Math.max(0d,avIntersect.getX())),
                            screen.heightPixels()*Math.min(1d,Math.max(0d,(1d-avIntersect.getY())))));
                }
            }
        }
        keyTap.set(false);
        GestureList gestures = frame.gestures();
        for (int i = 0; i < gestures.count(); i++) {
            if(gestures.get(i).type()==Gesture.Type.TYPE_KEY_TAP){
                keyTap.set(true); break;
            }
        }
    }
    
    private Vector Average(LimitQueue<Vector> vectors)
    {
        float vx=0f, vy=0f, vz=0f;
        for(Vector v:vectors){
            vx=vx+v.getX(); vy=vy+v.getY(); vz=vz+v.getZ();
        }
        return new Vector(vx/vectors.size(), vy/vectors.size(), vz/vectors.size());
    }
    
    private class LimitQueue<E> extends LinkedList<E> {
        private int limit;
        public LimitQueue(int limit) {
            this.limit = limit;
        }

        @Override
        public boolean add(E o) {
            super.add(o);
            while (size() > limit) { super.remove(); }
            return true;
        }
    }
}

And finally, here you can see it in action:

 

5. POC #3. JavaFX 3D and Leap, with JDK8

The last part of this post will cover my experiments with the recent early access releases of JDK8, build b92 on the time of this writing, as JavaFX 3D is enabled since b77. Here you can read about the 3D features planned for JavaFX 8. 

Installing JDK8 is easy, and so is creating a JavaFX scene with 3D primitives like spheres, boxes or cylinders, or even user-defined shapes by meshes, defined by a set of points, texture coordinates and faces.



There are no loaders for existing 3D file formats (obj, stl, Maya, 3D Studio, ...). So if you want to import a 3D model, you need one. 

The first place to start looking is in OpenJFX, the open source code home of JavaFX development.

You'll find in their repository between their experiments, as they call them, a 3D Viewer. So download it from their repository, build it and see what you can do!



For instance, you can drag and drop an obj model. The one in the picture is a model of a Raspberry Pi, downloaded from here.

For other formats not yet supported, you can go to InteractiveMesh.org, where August Lammersdorf has released several importers (3ds, obj and stl) for JDK8 b91+. Kudos to him for his amazing work and contributions!

I'll use his 3ds importer and the Hubble Space Telescope model from NASA, to add this  model to a JavaFX scene, and then I'll try to add touch-less rotation and scaling options.

First of all, we need a little mathematical background here, as rotating a 3D model in JavaFX requires a rotation axis and angle. If we have several rotations to make at the same time we need to construct a rotation matrix, and after that get the rotation axis and its angle.

As the Leap provides three rotations from a hand: pitch (around its X axis) , yaw (around its Y axis) and roll (around its Z axis), providing the model is already well orientated (otherwise we'll need to add previous rotations too), the rotation matrix will be: 

 where:
  
So:
  
Then, the angle and the rotation unitary axis components can be easily computed from:


Special care has to be taken when converting Leap roll, pitch and yaw angles values to those required for the JavaFX coordinate system (180º rotated from X axis).

With this equations, we just need to listen to hand rotation changes and compute the rotation axis and angle values on every change to rotate accordingly the 3D model. 

So now we're ready to try our POC 3D sample: import a 3ds model and perform rotations with our hand through the Leap Motion Controller.
 
The following code snippet shows how it is done for the JavaFX class:

public class JavaFX8 extends Application {
    private AnchorPane root=new AnchorPane();
    private final Rotate cameraXRotate = new Rotate(0,0,0,0,Rotate.X_AXIS);
    private final Rotate cameraYRotate = new Rotate(0,0,0,0,Rotate.Y_AXIS);
    private final Translate cameraPosition = new Translate(-300,-550,-700);
    private SimpleLeapListener listener = new SimpleLeapListener();
    private Controller leapController = new Controller();
    
    @Override 
    public void start(Stage stage){
        final Scene scene = new Scene(root, 1024, 800, true);
        final Camera camera = new PerspectiveCamera();
        camera.getTransforms().addAll(cameraXRotate,cameraYRotate,cameraPosition);
        scene.setCamera(camera);
        controller.addListener(listener);

        TdsModelImporter model=new TdsModelImporter();
        try {
            URL hubbleUrl = this.getClass().getResource("hst.3ds");
            model.read(hubbleUrl);
        }
        catch (ImportException e) {
            System.out.println("Error importing 3ds model: "+e.getMessage());
            return;
        }
        final Node[] hubbleMesh = model.getImport();
        model.close();
        final Group model3D = new Group(hubbleMesh);
 
        final PointLight pointLight = new PointLight(Color.ANTIQUEWHITE);
        pointLight.setTranslateX(800);
        pointLight.setTranslateY(-800);
        pointLight.setTranslateZ(-1000);
        root.getChildren().addAll(model3D,pointLight);

        listener.posHandLeftProperty().addListener(new ChangeListener<Point3D>(){
            @Override public void changed(ObservableValue<? extends Point3D> ov, Point3D t, final Point3D t1) {
                Platform.runLater(new Runnable(){
                    @Override public void run() {
                        if(t1!=null){
                            double roll=listener.rollLeftProperty().get();
                            double pitch=-listener.pitchLeftProperty().get();
                            double yaw=-listener.yawLeftProperty().get();
                            matrixRotateNode(model3D,roll,pitch,yaw);
                        }
                    }
                });
            }
        });
    }
    private void matrixRotateNode(Node n, double alf, double bet, double gam){
        double A11=Math.cos(alf)*Math.cos(gam);
        double A12=Math.cos(bet)*Math.sin(alf)+Math.cos(alf)*Math.sin(bet)*Math.sin(gam);
        double A13=Math.sin(alf)*Math.sin(bet)-Math.cos(alf)*Math.cos(bet)*Math.sin(gam);
        double A21=-Math.cos(gam)*Math.sin(alf);
        double A22=Math.cos(alf)*Math.cos(bet)-Math.sin(alf)*Math.sin(bet)*Math.sin(gam);
        double A23=Math.cos(alf)*Math.sin(bet)+Math.cos(bet)*Math.sin(alf)*Math.sin(gam);
        double A31=Math.sin(gam);
        double A32=-Math.cos(gam)*Math.sin(bet);
        double A33=Math.cos(bet)*Math.cos(gam);
        
        double d = Math.acos((A11+A22+A33-1d)/2d);
        if(d!=0d){
            double den=2d*Math.sin(d);
            Point3D p= new Point3D((A32-A23)/den,(A13-A31)/den,(A21-A12)/den);
            n.setRotationAxis(p);
            n.setRotate(Math.toDegrees(d));                    
        }
    }
}

And this code snippet shows how it is done for the Leap Listener class:

public class SimpleLeapListener extends Listener {
    private ObjectProperty<Point3D> posHandLeft=new SimpleObjectProperty<Point3D>();
    private DoubleProperty pitchLeft=new SimpleDoubleProperty(0d);
    private DoubleProperty rollLeft=new SimpleDoubleProperty(0d);
    private DoubleProperty yawLeft=new SimpleDoubleProperty(0d);
    private LimitQueue<Vector> posLeftAverage = new LimitQueue<Vector>(30);
    private LimitQueue<Double> pitchLeftAverage = new LimitQueue<Double>(30);
    private LimitQueue<Double> rollLeftAverage = new LimitQueue<Double>(30);
    private LimitQueue<Double> yawLeftAverage = new LimitQueue<Double>(30);

    public ObservableValue<Point3D> posHandLeftProperty(){ return posHandLeft; }
    public DoubleProperty yawLeftProperty(){ return yawLeft; }
    public DoubleProperty pitchLeftProperty(){ return pitchLeft; }
    public DoubleProperty rollLeftProperty(){ return rollLeft; }
    
    @Override
    public void onFrame(Controller controller) {
        Frame frame = controller.frame();
        if (!frame.hands().empty()) {
            Screen screen = controller.calibratedScreens().get(0);
            if (screen != null && screen.isValid()){
                Hand hand = frame.hands().get(0);
                if(hand.isValid()){
                    pitchLeftAverage.add(new Double(hand.direction().pitch()));
                    rollLeftAverage.add(new Double(hand.palmNormal().roll()));
                    yawLeftAverage.add(new Double(hand.direction().yaw()));                    
                    pitchLeft.set(dAverage(pitchLeftAverage).doubleValue());
                    rollLeft.set(dAverage(rollLeftAverage).doubleValue());
                    yawLeft.set(dAverage(yawLeftAverage).doubleValue());
                    
                    Vector intersect = screen.intersect(hand.palmPosition(),hand.direction(), true);
                    posLeftAverage.add(intersect);
                    Vector avIntersect=Average(posLeftAverage);
                    posHandLeft.setValue(new Point3D(screen.widthPixels()*Math.min(1d,Math.max(0d,avIntersect.getX())),
                            screen.heightPixels()*Math.min(1d,Math.max(0d,(1d-avIntersect.getY()))),
                            hand.palmPosition().getZ()));
                }
            }                
        }
    }
    private Double dAverage(LimitQueue<Double> vectors){
        double vx=0;
        for(Double d:vectors){
            vx=vx+d.doubleValue();
        }
        return new Double(vx/vectors.size());
    }
}

In the following video I've added a few more things, which aren't in the previous code: with the hand Z position we can scale the model, and we look for right hand circle gestures, to start an animation to rotate indefinitely the model, till another circle gesture is found, resuming hand rotations.



Conclusions

With these few samples I think you've already shown the impressive potential of a device like the Leap Motion Controller. 

JavaFX as RIA platform can interact with the Leap Motion device nicely and take UI to the next level.

We're all waiting eagerly to the public release of the device, and the opening of the Airspace market, where we'll find all kind of applications to use the Leap with.

This will change definitely the way we interact with our computers for ever.

Thank you for reading, as always, any comment will be absolutely welcome.