Mash, a library for concurrent applications

This Clojure/Clojurescript library aims to facilitate the development of concurrent programs. It is intended for decentralized, peer to peer applications. It provides a simple, practical syntax to define a system of small concurrent processes (similar to actors). The processes can interact by messaging or via shared variables.

Goals

Mash is an experimental, minimal framework. It evolved as part of a larger project, a prototype for a decentralized application.

There is already a number of libraries or languages to help the development of distributed systems. However, many of those systems do not run on the browser. More importantly, they are often focused on solving low-level problems. Mash uses publish-subscribe messaging, but does not provide an implementation.

Mash isn’t a product, or even a fully usable framework, but it demonstrates one possible design option to program decentralized applications.

In a nutshell

Mash is a small library, and can be fully explain in a few paragraphs.

Messaging

A Mash program runs on a number of nodes (ie separate instances). The program is collection of processes.

Nodes communicate via publish-subscribe messaging (“pub/sub”).

Messages have two parts : a topic and some content.

Topics are vectors of keywords or numbers :

[:foo 1 :bar :baz 23]

The content is a hashmap of keys and values :

{:foo "text" :bar 42 :baz 125}

Processes select messages they want to receive. They define their message filter by pattern matching on the topic. For instance, the pattern [:foo :* :bar :baz :*], contains the wildcard :*. would match the example topic above. Something like [:qux :*] would not.

State variables

The shared application state is held in a hashmap called sharedstate1. sharedstate is replicated across all nodes ; any change by a client is immediately mirrored to all other clients.

We call the content of this state hashmap the state variables : the value of a state variable var is the value for the key var in sharedstate.

Processes

In a Mash program, the basic building block is the process.

We use defp to define a new process 2 :

(defp pname [foo bar] topic
 ([{:arg1 42 :arg2 _ :arg3 var3}] 
    <body>)
    ...)

Where :

  • pname is a process name

  • foo, bar … are the arguments. They are bound in <body> to the current values of the state variables foo, bar

  • topic is the topic the process is subscribed to.

Processes are not invoked manually by the programmer ; instead, they are triggered by events. In other words, they run whenever a message with a topic matching topic is received by the node.

  • {:arg1 42 :arg2 _ :arg3 var3} :

The message content is pattern-matched via core.match. For instance, if the incoming message content is the map {:arg1 42 :arg2 54 :arg3 "example"}, the variable var3 will be bound to the value “example” in the <body>3.

Effects of a process:

The return value of <body> is ignored.

There are two ways for processes to have an effect:

  • Change the value of a state variable, with (write! var val).

  • Send a message, with (send topic message). For instance :

(send [:topic 1 :subtopic 42]
      {:arg1 "text-value" :arg2 123})

Using these two primitives, a process may send messages, and mutate variables. But those actions are not immediate. The messages and state updates will be queued up, and all sent together when <body> returns.

Example : chat

For a concrete, but minimalistic example of a tiny chat application, see below.

What’s it for ?

  • Lightweight and natural syntax in Clojure
  • Reduced mental overhead, compared with lower-level concurrency constructs (such as pure actors)
  • Serverless, apart from the message broker currently. Usable in decentralized / P2P systems, in principle
  • Runs in the browser and/or on the server
  • Works well with React, as the whole application state is easily accessed in a single object.

Notes

  • Implementation :

Typically, a process will send new messages and modify some shared state variables (and maybe some local private state). These actions are be cached, and later sent all at once, when the process ends. Shared state updates are just broadcast as messages, with a special topic.

Our Mash implementation relies upon a RabbitMQ message broker. Removing this broker will make the application fully decentralized. In principle, any other message broker could be used, or for that matter, any other topology. It appears that decentralized pub-sub systems exist today, but they are still a work-in-progress.

  • Distributed systems issues :

One of the hardest problems in this model is consistency : we want to ensure that all nodes process events in the same order (at least when it is essential), particularly under high loads. The most suitable implementation strategy for this is still an open question! Unfortunately, for now, it’s still partly the application programmer’s responsibility to ensure that conflicts can’t happen. To help with this, a decision was made to ensure processes always run sequentially, rather than in parallel.

  • Inspiration :

The idea of actors interacting in a multi-party communication model was taken from Marketplace by Tony Garnock-Jones. 4 The author’s PhD also provides an interesting introduction to the concept of Conversational Concurrency.

  • Miscellaneous :

Currently, there is no focus on performance, or scalability.

While this experiment was developed in the Clojure ecosystem, a Javascript implementation is envisaged for the future.

Mash is a prototype that was originally developed in the context of another project. The implementation isn’t open-source, but we hope others will find the idea interesting or useful.

Example Application

alt text

Here is one way to implement a very minimalistic chat.

Note: in this particular simple example, all the messaging is actually internal to each single node. Nodes communicate with each other via the shared variables only.

First, the processes :

(def *node-id* (rand-int 10000)) ; generate an unique identifier for this node

;; Triage my input : a "message" command to send out a new message,
;; or a "name" command to add this node as participant.
(defp input->command []
  [:input *node-id*]
  ([{:command command
     :node id
     :arg arg}]
   (case command
     "message"
     (send [:message *node-id*]
           {:message arg
            :from *node-id*
            :name @s/username})
     "name"
     (send [:participant *node-id*]
           {:name arg
            :id *node-id*}))))

;; Add ourselves to the state variable :participants.
(defp store-new-participant [participants]
  [:participant *node-id*]
  ([{:id id :name name}]
   (reset! s/username
           name)
   (write! :participants
           (vec (assoc (vec->map participants)
                       id name)))))

;; Add our latest message to the state variable :messages.
(defp store-new-message [messages]
  [:message *node-id*]
  ([{:message msg :from id}]
   (write! :messages
           (conj messages
                 [id msg]))))

;; Update a global var containing the last message. This is used in the UI.
(defp signal-new-message []
  [:message :*]
  ([{:message message :from id :name who}]
   (reset! s/latest
           [who message])))

;; This process is a bot. It responds "Good morning" to whoever sends "Hello"
(defp bot []
  [:message :*]
  ([{:message msg :from id :name name}]
   (when (and @s/bot?    ; check global variable : is bot enabled?
              name
              (re-seq #"Hello"
                      msg)
              (not= id
                    *node-id*))
     (go
       (<! (timeout 200))
       (send [:message *node-id*]
             {:message (str "Good morning, " name)
              :from *node-id*
              :name @s/username})))))

Here is the Reagent/React UI. (just add some CSS)

(defn InputArea [class field title command]
  [:div {:class class}
   [:input {:type "text"
            :value @field
            :on-change
            (fn [^js/SyntheticEvent v]
              (let [newval (.-value (.-target v))]
                (reset! field
                        newval)))}]
   [:div
    [:a
     {:on-click #(do
                   (input-callback [command @field]))}
     title]]])

(def ui-message (r/atom ""))
(defn MsgInput []
  [InputArea "class1" ui-message "Send" "message"])

(def ui-name (r/atom ""))
(defn NameInput []
  [InputArea "class2" ui-name "Edit name" "name"])

(defn Participants [names]
  [:div.participants-area
   [:p "Participants :"]
   [:ul
    (map (fn [name]
           [:li.pname name])
         names)]])

(defn BotControl []
  [:div#bot-control
   [:button
    {:on-click #(swap! s/bot?
                       not)}
    (str (if @s/bot? "Deactivate" "Activate") " Bot")]])
  
(defn ControlArea []
  (let [participants (vec->map (:participants @sharedstate))]
    [:div.control-area
     [:p (str "I am " (or @s/username "<unnamed>") ".")]
     [NameInput]
     [BotControl]]))

(defn MessageArea [messagelist]
  [:div.message-area
   [:table
    [:tbody
     (map (fn [[author msg]]
            [:tr
             [:td.author author]
             [:td.message msg]])
          messagelist)]]])

(defn TransientInfo []
  (when-let [[who msg] @s/latest]
    [:div#transient
     [:p (str who " wrote : " msg)]]))

(defn home-page []
  (fn []
    (let [participants (vec->map (:participants @sharedstate))
          names (vals participants)
          messages (:messages @sharedstate)]
      [:div
       [:div
        [ControlArea]
        [Participants names]
        [MessageArea (map (fn [[id msg]]
                            [(participants id) msg])
                          messages)]]
       [:div#bottom-area
        [MsgInput]
        [TransientInfo]]])))

Finally, a bit of input handling code. This converts input from the UI to messages.

(def input-chan (chan))

(def input-callback
  (fn [input]
    (go (>! input-chan
            input))))

(defn input-loop []
  (go-loop []
    (let [[command arg] (<! input-chan)]
      (send [:input *node-id*]
            {:command command
             :node *node-id*
             :arg arg}))
    (recur)))

Contact us

Please contact Pierre with any comments.

Citation style : Light Meta, “Mash, a library for concurrent applications” (2019).

(c) View Systems Ltd. All Rights Reserved.

  1. To be mutable, it is contained in a Clojure atom. 

  2. Note for non-Clojurians : this is a bit similar to a Clojure function definition. 

  3. _ is a core.match wildcard. 

  4. Garnock-Jones, T., Tobin-Hochstadt, S., Felleisen, M.: The network as a language construct. In: European Symp. on Programming. (2014) 473–492