View on GitHub

Jscex

Async flow control in JavaScript, with JavaScript

Download this project as a .zip file Download this project as a tar.gz file

中文用户请点此访问中文版

"Jscex" is short for "JavaScript Computation EXpressions". It provides a monadic extensions for JavaScript language and would significantly improve your programming life in certain scenarios. The project is written in JavaScript completely, which mean it can be used in any execution engines support ECMAScript 3, including mainstream browsers or server side JavaScript environments (e.g., Node.js).

Currently features:

Jscex asynchronous library

Asynchronous programming is essential, especially in JavaScript. JavaScript doesn't provide any primitives for writing codes can block the execution. Any operations which take a period of time have to be written in async ways. Async operations usually send back the results by callback functions when finished, and we could do the rest of work in the callback.

Here comes the problem. We usually express our work linearly, but async tasks with callbacks require logical division of algorithms. We cannot write conditions with if or loops with while/for/do statements. It's also very difficult to combine multiple asynchronous operations or handle exceptions and cancellation.

So Jscex comes to the rescue with its asynchronous library.

How to use

Node.js

If you want to use Jscex with Node.js, please install the following packages from npm:

npm install jscex jscex-jit jscex-async jscex-async-powerpack

then initialize these packages in your js file:

var Jscex = require("jscex");
require("jscex-jit").init(Jscex);
require("jscex-async").init(Jscex);
require("jscex-async-powerpack").init(Jscex);

You can also copy all the files under "src" folder to get the latest build.

Web page

If you want to use Jscex with in web page, you can references the files under "src" folder in the pge:

<!-- base -->
<script src="jscex.js"></script>

<!-- parser -->
<script src="jscex-parser.js"></script>

<!-- JIT -->
<script src="jscex-jit.js"></script>

<!-- async builder and powerpack -->
<script src="jscex-builderbase.js"></script>
<script src="jscex-async.js"></script>
<script src="jscex-async-powerpack.js"></script>

All the files list above are uncompressed version mainly used for developement. The minified version of them are in "bin" folder, which is suitable for production use (if you're not using AOT compile mode, see the "AOT compiler" section for more detail). Furthermore, the dev version of Jscex compiler would produce programmer-friendly code, which is easily for debugging and the minified version would run a little faster.

Define an async function

Now everything is prepared, here's how we compile a normal JavaScript function with async builder:

var somethingAsync = eval(Jscex.compile("async", function (a, b) {
    // implementation
}));

Please visit the following sections for more details.

Samples:

All the following samples can be found in samples/async folder.

Clock:

We are going to draw a clock with HTML5 canvas on the page (samples/async/clock.html). It's rather easy for most front-end programmers:

function drawClock(time) {
    // clear and canvas and draw a clock on it.
}

setInterval(1000, function () {
    drawClock(new Date());
});

That's the implementation with callback, but in Jscex we could write code like this:

var drawClockAsync = eval(Jscex.compile("async", function (interval) {
    while (true) {
        drawClock(new Date());
        // wait for an async operation to complete
        $await(Jscex.Async.sleep(interval));
    }
}));

drawClockAsync(1000).start();

We wrote an infinite loop in the async method drawClockAsync, in each iteration we draw a clock with current time and call Jscex.Async.sleep method, the sleep operation blocks the code for 1 seconds.

How can we block the code without the support of runtime? The magic here is: we are not actually executing the code we wrote, the Jscex.compile method accept the function we provide and convert it into another (it's not necessory for us to understand the code currently):

(function (interval) {
    var $$_builder_$$_0 = Jscex.builders["async"];
    return $$_builder_$$_0.Start(this,
        $$_builder_$$_0.Loop(
            function () {
                return true;
            },
            null,
            $$_builder_$$_0.Delay(function () {
                drawClock(new Date());
                return $$_builder_$$_0.Bind(Jscex.Async.sleep(interval), function () {
                    return $$_builder_$$_0.Normal();
                });
            }),
            false
        )
    );
})

The string form of the new function generated by Jscex compiler will be dynamicly executed by eval method, preserving the current scope and context (variables, closures, etc.).

The $await method is the "bind" operation of async builder, it tells the compiler to put the code after that in the callback for the builder's "Bind" method. The $await method accepts an async task generated by Jscex.Async.sleep, provides a semantic of "waiting the operation to complete". The Bind method also start the task if it's not running.

It seems the implementation with Jscex is a bit longer in this simple case - please look at the following samples. They will tell you the real power of Jscex.

Animations

Animations are important for rich user interfaces. Let's build an animation like "move the element from here to there in a period of time" (samples/async/move.html). The traditional version of the move could be:

var moveTraditionally = function (e, startPos, endPos, duration, callback) {

    var t = 0;

    // move a bit
    function move() {
        e.style.left = startPos.x + (endPos.x - startPos.x) * t / duration;
        e.style.top = startPos.y + (endPos.y - startPos.y) * t / duration;

        t += 50;
        if (t < duration) {
            setTimeout(50, move);
        } else { // finished
            e.style.left = endPos.x;
            e.style.top = endPos.y;
            callback();
        }
    }

    setTimeout(50, move);
}

Can someone tell me the algorithm used to move the element? Maybe we would understand the implementation after reading the code again and again, but it's really difficult and uncomfortable for the programmer to read and write codes like that. But everything would be changed with Jscex:

var moveAsync = eval(Jscex.compile("async", function (e, startPos, endPos, duration) {
    for (var t = 0; t < duration; t += 50) {
        e.style.left = startPos.x + (endPos.x - startPos.x) * t / duration;
        e.style.top = startPos.y + (endPos.y - startPos.y) * t / duration;
        $await(Jscex.Async.sleep(50));
    }

    e.style.left = endPos.x;
    e.style.top = endPos.y;
}));

We could express our algorithm in normal way (linearly): loop with a for statement, sleep for 50 milliseconds in each iteration and move the element again. It's just simple and elegant.

Now we got an async method moveAsync, we can use it when building another async method, just like the Jscex.Async.sleep method in the core library. They both return the async tasks implementing the same protocal for outside. So check out the method below:

var moveSquareAsync = eval(Jscex.compile("async", function(e) {
    $await(moveAsync(e, {x:100, y:100}, {x:400, y:100}, 1000));
    $await(moveAsync(e, {x:400, y:100}, {x:400, y:400}, 1000));
    $await(moveAsync(e, {x:400, y:400}, {x:100, y:400}, 1000));
    $await(moveAsync(e, {x:100, y:400}, {x:100, y:100}, 1000));
}));

We can easily find out how moveSquareAsync method works: it moves an element with a square routine. It's really easy to combine multiple async operations into another in Jscex.

Sorting animations:

Every programmer learns sorting algorithms, like bubble sort:

var compare = function (x, y) {
    return x - y; 
}

var swap = function (array, i, j) {
    var t = array[x];
    array[x] = array[y];
    array[y] = t;
}

var bubbleSort = function (array) {
    for (var x = 0; x < array.length; x++) {
        for (var y = 0; y < array.length - x; y++) {
            if (compare(array[y], array[y + 1]) > 0) {
                swap(array, y, y + 1);
            }
        }
    }
}

The basic idea of represent an sorting algorithm with animations is simple: repaint the graph after each swap and wait for a little time. But the implementation is not as easy as the idea looks like, we cannot use for loops anymore if we "stop" the code execution for some time using setTimeout. But let's check out the Jscex version:

var compareAsync = eval(Jscex.compile("async", function (x, y) {
    $await(Jscex.Async.sleep(10)); // each "compare" takes 10 ms.
    return x - y;
}));

var swapAsync = eval(Jscex.compile("async", function (array, x, y) {
    var t = array[x];
    array[x] = array[y];
    array[y] = t;

    repaint(array); // repaint after each swap

    $await(Jscex.Async.sleep(20)); // each "swap" takes 20 ms.
}));

var bubbleSortAsync = eval(Jscex.compile("async", function (array) {
    for (var x = 0; x < array.length; x++) {
        for (var y = 0; y < array.length - x; y++) {
            var r = $await(compareAsync(array[y], array[y + 1]));
            if(r > 0) {
                $await(swapAsync(array, y, y + 1));
            }
        }
    }
}));

I believe there's no need to explain more - it's just the standard "bubble sort" algorithm. And of course we can create animation for "quick sort":

var _partitionAsync = eval(Jscex.compile("async", function (array, begin, end) {
    var i = begin;
    var j = end;
    var pivot = array[Math.floor((begin + end) / 2)];

    while (i <= j) {
        while (true) {
            var r = $await(compareAsync(array[i], pivot));
            if (r < 0) { i++; } else { break; }
        }

        while (true) {
            var r = $await(compareAsync(array[j], pivot));
            if (r > 0) { j--; } else { break; }
        }

        if (i <= j) {
            $await(swapAsync(array, i, j));
            i++;
            j--;
        }
    }

    return i;
}));

var _quickSortAsync = eval(Jscex.compile("async", function (array, begin, end) {
    var index = $await(_partitionAsync(array, begin, end));

    if (begin < index - 1) {
        $await(_quickSortAsync(array, begin, index - 1));
    }

    if (index < end) {
        $await(_quickSortAsync(array, index, end));
    }
}));

var quickSortAsync = eval(Jscex.compile("async", function (array) {
    $await(_quickSortAsync(array, 0, array.length - 1));
}));

The complete sample is in "samples/async/sorting-animations.html". After opening the page with browser support HTML5 canvas, you can click the links to view the animation of three sorting algorithms: bubble sort, selection sort and quick sort.

Tower of Hanoi

Tower of Hanoi is a puzzle of moving discs. It has a simple, recursive solution:

  1. move n−1 discs from A to B (with the help of C). This leaves disc n alone on peg A
  2. move disc n from A to C
  3. move n−1 discs from B to C (which the help of A) so they sit on disc n

which in code:

var hanoi = function (n, from, to, mid) {
    if (n > 0) hanoi(n - 1, from, mid, to);
    moveDisc(n, from, to);
    if (n > 0) hanoi(n - 1, mid, to from);
}

hanoi(5, "A", "C", "B");

If we need to show the algorithm in animation (samples/async/hanoi.html), we could re-write the code just like the sample above:

var hanoiAsync = eval(Jscex.compile("async", function(n, from, to, mid) {
    if (n > 0) {
        $await(hanoiAsync(n - 1, from, mid, to));
    }

    $await(moveDiscAsync(n, from, to));

    if (n > 0) {
        $await(hanoiAsync(n - 1, mid, to, from));
    }
}));

hanoiAsync(5, "A", "C", "B").start();

If you open the sample page in the browser, you'll find the discs is moving one by one, resolving the puzzle automatically. Maybe it's not quite easy for someone to follow each step, so let's just make a little change:

var hanoiAsync = eval(Jscex.compile("async", function(n, from, to, mid) {
    if (n > 0) {
        $await(hanoiAsync(n - 1, from, mid, to));
    }

    // wait for the button's being clicked
    var btnNext = document.getElementById("btnNext");
    $await(Jscex.Async.onEvent(btnNext, "click"));

    $await(moveDiscAsync(n, from, to));

    if (n > 0) {
        $await(hanoiAsync(n - 1, mid, to, from));
    }
}));

Before each moveDiscAsync operation, the program would wait for the button's "click" event. In Jscex async library, "async task" only means "operation would be finished in the future". In the example above, the button's "click" event is also an async task - the task would be finished when the button is clicked, then the program keeps going, move a disc and wait for another click.

People can write async programs without callbacks, that's how Jscex improve productivity and maintainability.

Simple web-server with Node.js

Jscex works for any execution engines support ECMAScript 3, not only browsers but also server-side environment like Node.js. Node.js is a server-side JavaScript environment that uses an asynchronous event-driven model. This allows Node.js to get excellent performance based on the architectures of many internet applications.

Here's a simple file server build with Node.js:

var http = require("http");
var fs = require("fs");
var url = require("url");
var path = require("path");

var transferFile = function (request, response) {
    var uri = url.parse(request.url).pathname;
    var filepath = path.join(process.cwd(), uri);

    // check whether the file is exist and get the result from callback
    path.exists(filepath, function (exists) {
        if (!exists) {
            response.writeHead(404, {"Content-Type": "text/plain"});
            response.write("404 Not Found\n");
            response.end();
        } else {
            // read the file content and get the result from callback
            fs.readFile(filepath, "binary", function (error, data) {
                if (error) {
                    response.writeHead(500, {"Content-Type": "text/plain"});
                    response.write(error + "\n");
                } else {
                    response.writeHead(200);
                    response.write(data, "binary");
                }

                response.end();
            });
        }
    });
}

http.createServer(function (request, response) {
    transferFile(request, response);
}).listen(8124, "127.0.0.1");

There're two async method used above: path.exists and fs.readFile. Most I/O api in Node.js are asynchronous, which brings great scalability but low programmability. But with the help of Jscex, we can write async programs as easy as normal code (samples/async/node-server.js):

require("../../lib/uglifyjs-parser.js");
require("../../src/jscex.js");
require("../../src/jscex.async.js");
require("../../src/jscex.async.node.js");

Jscex.Async.Node.Path.extend(path);
Jscex.Async.Node.FileSystem.extend(fs);

var transferFileAsync = eval(Jscex.compile("async", function (request, response) {
    var uri = url.parse(request.url).pathname;
    var filepath = path.join(process.cwd(), uri);

    var exists = $await(path.existsAsync(filepath));
    if (!exists) {
        response.writeHead(404, {"Content-Type": "text/plain"});
        response.write("404 Not Found\n");
    } else {

        try {
            var data = $await(fs.readFileAsync(filepath));
            response.writeHead(200);
            response.write(data, "binary");
        } catch (ex) {
            response.writeHead(500, {"Content-Type": "text/plain"});
            response.write(ex + "\n");
        }
    }

    response.end();
}));

http.createServer(function (request, response) {
    transferFileAsync(request, response).start();
}).listen(8125, "127.0.0.1");

All the Jscex files are compatible with CommonJS module specification, which can be loaded by require method in Node.js. Jscex async library has a simple model of "async tasks", everyone can easily write extensions/adaptors for existing async operations. Please check the following section for more details.

Limitations:

There're three limitations of the current version of Jscex - none of them becomes a real problem in my experiences.

Need separate $await statement

Jscex compiler can only handle explicit $await operation:

Other kinds of usages could not be compiled:

f(g(1), $await(...))

if (x > y && $await(...)) { ... }

We should put $await in separate statement:

var a1 = g(1);
var a2 = $await(...);
f(a1, a2);

if (x > y) {
    var flag = $await(...);
    if (flag) { ... }
}
Nested Jscex functions

If you write nested Jscex functions, the inner function can also be compiled with outside one. For example:

var outerAsync = eval(Jscex.compile("async", function () {

    var innerAsync = eval(Jscex.compile("async", function () {
        // inner implementations
    }));

}));

At runtime, outerAsync would be compiled to:

(function () {
    var $$_builder_$$_0 = Jscex.builders["async"];
    return $$_builder_$$_0.Start(this,
        $$_builder_$$_0.Delay(function () {

            var innerAsync = (function () {
                var $$_builder_$$_1 = Jscex.builders["async"];
                return $$_builder_$$_1.Start(this,
                    // compiled inner implementations
                    $$_builder_$$_1.Normal()
                );
            });

            return $$_builder_$$_0.Normal();
        })
    );
})

But the compilation (of inner function) is based on static code parsing and generation, so the compiler can only recognize the "standard pattern": eval(Jscex.compile("builderName", function () { ... })). Please see the section "AOT Compiler - Limitation" for more details.

Language support:

Jscex compiler could generate code for almost all the features of ECMAScript 3 except:

AOT compiler

The AOT (ahead-of-time) compiler is a piece of JavaScript code (scripts/jscexc.js) which generates code before runtime.

Why we need an AOT compiler.

The JIT compiler works just fine. The function would be compiled only once for each page load or node.js execution, so the performance cost of compiler is really small. And the size of compiler is only around 10K when minified and gzipped - acceptable for me.

But the problem comes with "script compressing". Normally, the script used for websites would be compressed by tools like UglifyJS, Closure Compiler or YUI compressor. Jscex compiler works fine when the compress strategy is just removing the whitespaces and shortening the name of variables, but modern compressors would rewrite the statement structures for getting minimal size:

var bubbleSortAsync=eval(Jscex.compile("async",function(a){for(var b=0;b<a.length;b++)for(var c=0;c<a.length-b;c++){var d=$await(compareAsync(a[c],a[c+1]));d>0&&$await(swapAsync(a,c,c+1))}}))

The code above is the code of bubble sort animation compressed by UglifyJS. Please notice that Jscex cannot handle code like d>0&&$await(...), so we have to compile the code before compressing. That the main reason I build an AOT compiler.

Usage

The AOT compiler runs with node.js:

node scripts/jscexc.js --input input_file --output output_file

For the bubbleSortAsync method above, it would be compiled into:

(function (array) {
    var $$_builder_$$_0 = Jscex.builders["async"];
    return $$_builder_$$_0.Start(this,
        $$_builder_$$_0.Delay(function () {
            var x = 0;
            return $$_builder_$$_0.Loop(
                function () {
                    return x < array.length;
                },
                function () {
                    x++;
                },
                $$_builder_$$_0.Delay(function () {
                    var y = 0;
                    return $$_builder_$$_0.Loop(
                        function () {
                            return y < (array.length - x);
                        },
                        function () {
                            y++;
                        },
                        $$_builder_$$_0.Delay(function () {
                            return $$_builder_$$_0.Bind(compareAsync(array[y], array[y + 1]), function (r) {
                                if (r > 0) {
                                    return $$_builder_$$_0.Bind(swapAsync(array, y, y + 1), function () {
                                        return $$_builder_$$_0.Normal();
                                    });
                                } else {
                                    return $$_builder_$$_0.Normal();
                                }
                            });
                        }),
                        false
                    );
                }),
                false
            );
        })
    );
})

The AOT compiler would keep the code others than Jscex functions. The code generated by AOT compiler could be compressed safely. Futhermore, the compiled code could execute without "jscex-parser.js" and "jscex-jit.js". The async methods could be executed properly with only "jscex-async.js" and "jscex-async-powerpack", which is only 3KB when gzipped.

Limitation

The AOT compiler would parse the input scripts statically and generate new code, so it can only recognize the standard pattern: eval(Jscex.compile("builderName", function () { ... })). The follow codes work fine with JIT compiler but cannot be compiled by AOT compiler.

var compile = Jscex.compile;
var builderName = "async";
var func = function () { ... };
var newCode = compile(builderName, func);
var funcAsync = eval(newCode);

Luckily, the standard pattern is quite enough in my experiences, so this limitation won't be an issue in real world.

Beyond async

Jscex is not only for async programming. The Jscex compiler turns the input function into a standard monadic form, the rest work are done by "Jscex builder". Jscex releases with a build-in "async builder" could simplify async programming, we can also define a "seq builder" to help constructing "iterators" in JavaScript (ECMAScript 3) - like the "generator" feature in JavaScript 1.7 or Python/C#, etc.

var rangeSeq = eval(Jscex.compile("seq", function (minInclusive, maxExclusive) {
    for (var i = minInclusive; i < maxExclusive; i++) {
        $yield(i);
    }
}));

Jscex builders register themselves in Jscex.builders dictionary. The builder would be retrieved by name at runtime, you can get the following code by watching the console.log output or use the AOT compiler:

(function (minInclusive, maxExclusive) {
    var $$_builder_$$_0 = Jscex.builders["seq"];
    return $$_builder_$$_0.Start(this,
        $$_builder_$$_0.Delay(function () {
            var i = minInclusive;
            return $$_builder_$$_0.Loop(
                function () {
                    return i < maxExclusive;
                },
                function () {
                    i++;
                },
                $$_builder_$$_0.Delay(function () {
                    return $$_builder_$$_0.Bind(i, function () {
                        return $$_builder_$$_0.Normal();
                    });
                }),
                false
            );
        })
    );
})

Unfortunately, the "seq builder" is not part of Jscex at this moment. It's one of the future plans of Jscex project.

Comparison to other projects

There're several other projects has the same purpose. We can put these project into two categories.

Library extensions:

Most projects build libraries to simplify async programming in JavaScript:

// async.js (http://github.com/fjakobs/async.js)
async.list([
    asncFunction1,
    asncFunction2,
    asncFunction3,
    asncFunction4,
    asncFunction5,
]).call().end(function(err, result) {
    // do something useful
});

// Step (http://github.com/creationix/step)
Step(
    function readSelf() {
        fs.readFile(__filename, this);
    },
    function capitalize(err, text) {
        if (err) throw err;
        return text.toUpperCase();
    },
    function showIt(err, newText) {
        if (err) throw err;
        console.log(newText);
    }
);

People use these solutions need to follow the programming patterns defined by the library, but Jscex just follows JavaScript idioms, programming with Jscex is just programming in JavaScript. So Jscex has a really gentle learning curve.

Language extensions:

A few projects are "Language extensions" - for these projects like StratifiedJS and Narrative JavaScript, people write "JavaScript-like" codes and compile them to "real JavaScripts":

// Narrative JavaScript code
function sleep(millis) {
    var notifier = new EventNotifier();
    setTimeout(notifier, millis);
    notifier.wait->();
}

// StratifiedJS code
var result;
waitfor {
  result = performGoogleQuery(query);
}
or {
  result = performYahooQuery(query);
}    

Although these languages could be quite similar to JavaScript, but they're really not, especially the syntax and semantic related to async programming. Programmers have to learn "new languages" when using them. And these new languages may also break the JavaScript editors. Jscex is nothing more than JavaScript and even the $await operations are simple method calls - the only semantic related to that is "wait until finished", everyone knows how to programming in Jscex in minutes.

And in traditional JavaScript programming, people modify the code and refresh the page to see whether things got right. But these "new languages" cannot being executed in browsers (or other ECMASCript 3 engines) directly, it should be compiled into JavaScript before runtime. Project like Narrative and StratifiedJS also provide JIT compiler which load and produce code as runtime. But the way they use have obvious limitations:

If the sources are loaded by XMLHttpRequest, these source files have to be host in the same domain with website. JSONP can be used to load sources from other domains, but it's not loading remote sources directly, it loads the content in "method call" style - the source files have to be processed before sending to the client.

These project may also load script blocks written inside the page. The compiler would load these code snippets from the DOM and compile them when page loaded. But in the case like:

<!-- codes to compile -->
<script type="some-special-type">
    // define an async method
</script>

<script type="text/javascript">
    // cannot use the async method here
</script>

People cannot use the async methods defined in previous. It's not the behavior people are currently using in JavaScript programming.

But Jscex is nothing more than JavaScript:

Jscex just keeps nearly everything as usual for JavaScript programmers.

Jscex internals

(more details in the future)

Futures

Related projects

Bugs or Feedbacks?

Feel free to contact me for any bugs or feedbacks, please use the Google Groups or email me directly.

License

Jscex is released under the BSD license:

Copyright 2011 (c) Jeffrey Zhao <[email protected]>
Based on UglifyJS (http://github.com/mishoo/UglifyJS).

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

    * Redistributions of source code must retain the above
      copyright notice, this list of conditions and the following
      disclaimer.

    * Redistributions in binary form must reproduce the above
      copyright notice, this list of conditions and the following
      disclaimer in the documentation and/or other materials
      provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER “AS IS” AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.