JLOUIS Ramblings: 10/01/2011

Oct

23

Using the Event Tracer tool set in Erlang
A common problem in concurrent systems is the following: events fire from all places all the time and you have no direct control over when and why events fire. Thus, to figure out problems one can often make use of a sequence diagram like this one (thanks Wikipedia):

The purpose of such a diagram is to show the interactions between different concurrent parties. In this case, Fred, Bob, Hank and Renee - in a restaurant. One can easily draw such diagrams on paper. But the problem with such a drawing is that it is disconnected from the machine. What the drawing shows may or may not be what the program actually does.

Wouldn't it be neat if you could make such diagrams by tracing what the code does and then build up the diagram in a programmatic way? Well, with Erlang, you can. I'll use some code I wrote as an example for how to do it.

Step 1: Build a tracer function
The first step is to construct a tracer function. Mine look like the following and resides in a module called "utp".
```
report_event(DetailLevel, FromTo, Label, Contents) ->
    %% N.B External call
    ?MODULE:report_event(DetailLevel, FromTo, FromTo, Label, Contents).

report_event(_DetailLevel, _From, _To, _Label, _Contents) ->
    hopefully_traced.
```
The basic idea is found in the latter of these two definitions. It defines a dummy-function taking 5 arguments which are thrown away promptly. The return value "hopefully_traced" is arbitrary, but it signifies what we want out of this function. The first variant, is used when it is the same party that does both things.

The arguments are as follows:
- DetailLevel: A number between 0 and 100. It signifies the level of detail for this event. We can use it to give major events low level and more detailed event at a finer grained scale higher levels. The event tracer can cut off events based on this value so we get the desired graininess.
- From: The party from which the event originates.
- To: The party to which the event is sent.
- Label: The notational label to put on the message.
- Contents: If you click on the event, we will show this term to the user. It can be used to pack up more detailed information to present, while avoiding cluttering up the diagram.
Step 2: Trace the events in the program
Whenever something important in the program happens, you add a call to "utp:report_event/5" or whatever the designation for your tracing function is. This will be default do nothing, but it allows us to hook that function later with Erlangs tracing facilities. As an example, here is the interaction from above:
```
trace_test() ->
    Events = [{fred, bob, order_food},
              {bob, hank, order_food},
              {bob, fred, serve_wine},
              {hank, bob, pickup},
              {bob, fred, serve_feed},
              {fred, renee, pay}],
    [utp:report_event(50, F, T, L, [])
     || {F,T,L} <- Events].
```
But note that anything can be reported really, by spreading the event reporting function out over your code base. Rather than run scores of debug statements, you can add a reporter at the point in the code. Also note I have condensed it a bit in the above example with a list comprehension because I don't care for the detail level and the contents.

Step 3: Introduce code to invoke the et application
Next, we need to invoke Erlangs et application in the right way. I have a module, called "utp_filter" which does it, even though it still doesn't contain any filter function. Here is the code of interest:
```
start(ExtraOptions) ->
    Options =
        [{event_order, event_ts},
         {scale, 2},
         {max_actors, 10},
         {detail_level, 90},
         {actors, [fred, bob, hank, renee]},
         {trace_pattern, {utp, max}},
         {trace_global, true},
         {title, "uTP tracer"} | ExtraOptions],
    et_viewer:start(Options).
```
What this code does is to initalize the et_viewer in a way such that it can be used with our example. The event order is event_ts which means we are timestamping events at when they fire and not when they are received. The actors define the order in which the actors appear. The trace_pattern is pretty important. It is what makes us hook to the "utp:report_event/5" call. You can give a lot of options to this or event produce such patterns yourself. But just name the module in which your trace function is placed will work.

Step 4: Test
If we invoke "utp_filter:start([])" which starts up the et viewer by executing the code in step 3, we can then invoke the trace test code from step 2 to obtain:

Which should reflect the example from Wikipedia, but now obtained by executing a program in Erlang. This idea has been used by me to capture and find bugs in TCP-stack variants, in particular uTP:

(Even if these diagrams are small, the basic idea should be in there - it is not about the diagrams, but about the steps to obtain the glory yourself :). For a TCP stack-like protocol it is very powerful: You have a tcpdump(1) output, an strace(1) output, as well as the internal protocol state in the same diagram. I found quite some bugs with these tools in my code, simply by inspecting the interaction of the programs.

And thats it. With the above code in your application, you can enable a graphical trace viewer on your code at any time. If you don't enable it, the overhead of the trace functions are negligible - unless you have a very hot critical path indeed. If you do enable it, you can see exactly what your code does in the given situation.
Posted 23rd October 2011 by Anonymous

Labels: debugging erlang event tracing test tracing

7
View comments
Oct

10

Musings on Dart

Now the Dart language is out at http://dartlang.org, we can begin looking at the language and see what we think of it. Everything here are musings from reading the spec of the Dart language. Note that my points here are quite subjective at times. Don't expect this to be an objective run down of the language. Rather, I wanted to write down what I thought about the language just by looking at the specification alone.

Starting out, we generally won't focus on the syntax of the language. The syntax is boring, tedious and just there. It is Java/Javascript-like, mostly to interest the right kinds of people on the language. We note that the syntax is considerably shorter and leaner than the equivalent Java code and that there are notational
stuff going on to make certain things easier to write down in the programs.

The major goal of Dart is to clean up Javascript and make design choices in the language which does not hamper raw execution speed. Javascript is not built for compilation into fast programs. There are simply too many nasty warts and design choices in the language which makes it very hard to compile. V8 is astoundingly good at it, but by altering the source language a little bit, you can squeeze out much more power with very little effort.

I think that this is a goal of Dart which is central. Don't design yourself into a corner by language features.

Dart is Single Threaded but support concurrency through an Actor-like concept of isolates. The choice of making the runtime single threaded is understandable. It is vastly simpler to write a compiler for a single threaded runtime than it is for a multi-threaded one. Also, the focus is perhaps on writing programs for web browsers and the amount of extra processing power to be had there from multiple processors is perhaps limited. Note that while the language specification says it is single threaded, the Isolate concept in the library talks about heavy isolates which are separate threads. One could imagine that the execution inside an isolated region is always single-threaded (Like in, Erlang for instance), but the system as a whole might have several isolates working simultaneously.

On the other hand, even your modern mobile device goes multi-core. Thus, the choice is as much as a gamble as opting for a multi-core solution from day one. This choice is the conservative one.

Dart makes it explicit that it has several kinds of errors and that these may come "lazily" on demand. The idea is to facilitate JIT-compilation, so an implementation is not required to report all errors up front. An understandable and interesting choice. For a programmer with the proactive view (Haskell, Java, Ocaml) on error reporting, this sounds rather bad, but for a programmer with an reactive view (Erlang), it sounds quite familiar.

Naturally, the language makes the almost ubiquitous billion dollar mistake of C.A.R Hoare: the language includes null as a default. Almost all non-functional languages do.

An interesting choice is to enforce simplicity on static/final values. This is to ensure fast startup times, so a program can't hide a large computation in a global variable at startup.

Amusingly, the language must make sure that every function returns 'null' in the case where the function has no return statement. This is funny because in functional languages, this kind of problem doesn't exist. a 'void' function can't be used in expression context in an imperative language, but since Dart type system is not enforcing, there are no guarantees. Hence a kludge is introduced.

Functions are first class values with a simple notation for the equivalent of a lambda construction.

The language has operator overloading, but as is common in imperative languages, you cannot define your own operator. Rather, you must use the operators that come with the language.

Stealing ideas from other languages, there are convenience notation for getters and setters. On the other hand, there is no mention of lenses. The language is pretty much a standard OO-language with no mention of prototypical OO as in Javascript. This choice is probably smart since prototypical OO is irritating to compile.

There is a built-in notion of factory construction, much like the notation of getters and setters. The idea is to allow a constructor to act as a factory under the hood so the callers don't need to get exposed to the fact that there is a factory. One could imagine this concept extended to other classic design patterns over time.

The language is single-inheritance+interface+generics. Unlike Go, this language does support generics with upper bounds on the class hierarchy. In other words, the language has bounded quantification. And it even support F-bounded quantification on generics (also called recursively bounded quantification - you allow the class parameter T to occur in the bound as well: "T extends I(T)", see wikipedia on bounded quantification).

The language is pure. Everything is an object. You see that choice since null is the sole instance of class Null for example.

Dart has true integers of unbounded size. There is no cramming everything into double values. There are no pesky size bounds to battle. In other words, Dart picked the innovative choice.

I wonder... if a bool is an object, can I have a null bool?

Strings are unicode. We got lists and maps in the language. We have a functional expression notation for fast notation of small simple function evaluation thingies. It is perhaps the easiest way to get some functional ideas into a language without making it not seem like Java.

Note that a simple expression like '2 + 1' is internally compiled into a method invocation of the 2 objects + method with a parameter of 1.

In a very controversial move the type system is unsound. Having an unsound type system means that the type system may allow for certain bugs to happen in production code. Time will show if this is a problem in practice. It may be you almost never hit the problem in practice and then this choice might end up being pretty sane. But if you can hit the unsoundness easily, this choice will end up being pretty bad.

As in Java, constructing new types is had by making classes. To construct compound types, you need to use generics and parametrize the types as holes in the generic.

The most interesting part of the language, which is pretty conservative, is the choice of baking in both dynamic and static types into the same language. While the static type system is unsound, time will show if this unsoundness is troubling. C is also pretty bad type-wise, but that has not withheld people from writing software in it and using the type system to weed out bugs.

The other interesting part is hidden in the library as a library extension and are called Isolates. An Isolate is, technically, an actor which processes by itself. There is a concept of message passing ports between actors so one can send and receive information. Naturally, futures also form part of this definition.

My primary critique of the language is that Dart provides nothing new to the table. It looks like a Java-for-the-web clone without providing something new. Granted, it may sway some Java programmers to move toward the Web in the long run. Granted, this is in the interest of Google because the reliance on Java is dangerous when Oracle controls it. As a language, Dart is considerably less innovative than Google Go. Rather it buys into existing ideas, and adds a gradual type system as the most innovative feature. I am not too worried about the unsoundness currently, but it is an inherent danger lurking until the static checker will warn about it.

The secondary critique is the Isolate construction. First, there is a difference between a heavy and a light isolate. One provides a new thread of execution while the other provides an light-weight process in the current thread. There is no concept of migration so the programmer has to know beforehand how the program will run and execute to use isolates. It does provide pure isolation though, something not present in Go.

There are problems with this though. Can I create 100000 heavy isolates? Can I create 100000 light isolates? Suppose one of the light isolates ties up the CPU for a long time. Will isolates then be preemptively scheduled out so others gets a chance, or will the system keep running the single isolate while all others wait in queue? I am asking because it hurts latency.

Second, the notation is not built-in but provided as a library. This usually leads to code with heavier weight.

Third, there is no way to scrutinize the mailbox. You get a callback whenever a message is delivered, but it is up to you to build the search infrastructure for the mailbox. This is in strong contrast with Erlang and it usually leads to more complex actors in the long run.

Fourth, there is no concept of robustness. I can't link together Isolates so I can be told when other Isolates die and take care of it. Without links and monitors, it is cumbersome to build anything resembling OTP for Dart.

Fifth, where is my selective receive? We can easily imagine a single isolate having access to multiple ReceivePorts.

In conclusion, the current Isolate-implementation leaves much to be desired. I'd much rather want to hack Google Go in the browser than Dart to be honest. Go is a neater language, is more innovative, is farther ahead and has built-in-concurrency primitives. You just need to add a jQuery-like interface to Go and then it can substitute Dart as an imperative language.

What Dart DOES provide however is the ability to run faster than Javascript. It is clear that some of the choices enables the language to be compiled to much faster executing code. Also, if all you know is Javascript, the language is probably pretty neat. To the Java programmer, proper closures will be a warm welcome.

But for a guy who has seen things you wouldn't believe. Haskell ships off the shoulder of MultiCore CPUs. Watched OCaml glitter in the dark near TAPL. All those moments will be lost in time, like tears in rain. Time to die.

Finally, a disclaimer: I may be wrong on several occasions. I've had some 1-2 hours to study the language, and you can't really grasp a programming language in less than 1-2 months. So I may be wrong. Wrong on all points. Like fine wine, languages need a bit of time to mature and bloom into what they are. Dart is pretty young and drafty in its nature. A lot of things can happen.

Posted 10th October 2011 by Anonymous

Labels: concurrency dart language design programming languages rant

10
View comments
Oct

2

One major difference - ZeroMQ and Erlang.

When a dog owner wants to train his dog, the procedure is well-known and quite simple. The owner runs two loops: one of positive feedback and one of negative ditto. Whenever the dog does something right, the positive feedback loop is invoked and the dog is treated with a snack. Whenever the dog does something wrong, the dog is scolded and the negative feedback loop is used.

The result is positive and negative reinforcement of the dogs behavior. The dog will over time automatically behave as the owner wants and never even think of misbehaving.

When a programming language trains its leashed programmer, it likewise uses positive and negative feedback. Whenever a problem is easily solvable in the constructs and features of said language, it reinforces the use of those features and constructs. And also in the same vein, if something is hard to do in the language, the programmer will shy away from thinking the idea, since it may be too hard to do in the language. Another negative feedback loop is when resource usage of a program is bad. Either it will use too much memory of too many CPU resources to carry out its work. This discourages the programmer from using that solution again.

The important point is that while all practical general purpose languages are Turing complete, the way they train programmers to behave as they want is quite different. In an Object Oriented language for instance, the programmer is trained to reframe most - if not all - questions as objects of bundled fields and methods. A functional programmer is trained to envision programs as transformations of data from the form X into the form Y through the application of a function. And so on.

Which brings us to how ZeroMQ and Erlang will train its programmers into doing vastly different things:

In ZeroMQ, the programmer will configure a messaging network. That network will probably be fairly static. It is not the case that there is one configuration, invoked in the beginning of the system, which stays there for all of the systems execution, but chances are that the configuration is still fairly rigid and not subject to much change.

Around this messaging network, we now add processes which operate on the network. We send and receive messages on the network, processing them in various different programming languages. A process is also fairly static in the sense that the process is started, and then tend to use the messaging network as a queue: It will pick off some work, look at the data, process the data and push it back into the messaging network again. Taken to the extreme, ZeroMQ applications are often queue processors like this.

Incoming requests are transformed to a message and then sent into the messaging network of ZeroMQ. Then it will be processed by some vacant process in the other end and it will be moved from queue to queue until it finally completes. Note ZeroMQ is more complex than a simple point-to-point messaging framework, but the gist of the idea here is that many applications will have the above mentioned system design.

In Erlang, the programmer will take the request and turn the request into a process. Erlang programmers are trained by the language to like having a large amount of small processes like this, because the language encourages that behaviour. ZeroMQ-solutions on the other hand will be discouraged to dequeue a message and create a process out of it - if the processing language uses a traditional heavyweight thread for a process.

Now, given that the request is a process, an Erlang programmer will do the dual to the ZeroMQ programmer. Since the process is now the request, whenever the request needs to do some processing, it will do so itself. If it needs some external information, the request/process will ask for it. The preemptive scheduler will ensure that no request can stall the pipeline. Also, since the scheduler in Erlang is parallel, it is relatively easy to get all cores to work, but by having enough requests/processes in the run-queue.

Note, also, how the messaging network in this solution is fairly dynamic. Point-to-point exchanges are formed on demand when a request need some information.

To the Erlang programmer, the representation of the request is the process. There is no "message" living in a queue which is the representation of the request at any point. There are less need to tune fan-in/fan-out at the end of queues to make parallel work go through, since the schedulers will automatically parallelize.

Both styles have their advantages and disadvantages. But that is not the interesting point. The interesting point is how the ZeroMQ idea encourages one system design, whereas the Erlang solution encourages another. In Erlang a process is cheap, so it is totally feasible to turn each request into one. Furthermore, since there is no concept of a message channel, sending a message to a target is all about knowing or finding the name of the target (see for instance Ulf Wiger/Erlang Solutions gproc framework for an efficient name registry). On the contrary, since ZeroMQ has the concept of a generalized socket as a channel endpoint and assuming the programming language we use for processing does not have cheap processes. Then the only feasible solution is to build a static channel network with processors.

Another interesting point is that most languages with static type systems have latched onto the channel solution. It is easy because a channel can be annotated with the type traversing the channel. But it also trains programmers to think in that model. It would be fun to see a language where process Id's - that is names of processes - have a type of what that process receives.

The concluding remark: ZeroMQ/Erlang above is just an example. You can find this kind of reinforcement with basically any language design choice. It is rarely that a feature is ubiquitously good or bad. Rather, most choices are a trade-off. Haskells lazy evaluation enables some optimizations but disallow some other optimizations. Ocaml, being eager, can do the optimization that Haskell can't and can't do the ones that Haskell can. Similarly, for a programmer, getting easier access to some feature often means less control of something else, or that something else become much harder to pull off. The next time you enter in a discussion on the merits of a given language feature, make sure there is not a programming language holding your leash and telling you what is right or wrong!

Posted 2nd October 2011 by Anonymous

Labels: erlang programming languages rant zeromq

5
View comments

JLOUIS Ramblings

Computer Science, Mathematics, Society, Relationships, Sex, 0xf00d, Pets, Leashes and Secretaries.

Using the Event Tracer tool set in Erlang

View comments

Musings on Dart

View comments

One major difference - ZeroMQ and Erlang.

View comments