An experiment to replace Node-REDs existing NodeJS backend with an Erlang equivalent that is 100% compatible[1] to existing flow code.
The goal is bring the advantages of low-code visual flow-based programming to a programming language that is designed for message passing and concurrency from the ground up, hence Erlang.
[1] = 100% won’t be possible since function nodes that are coded in Javascript aren’t supported (or aren’t intended to be supported – unless someone has a workaround)
Node-RED is a amazing[*] tool for creating flows that describe concurrent processing, it is just a shame the NodeJS is single threaded. So why not use something that is multi-process from the ground up? Concurrency is guaranteed and included.
Also Erlang isn’t the most understandable of programming language – unless one has fallen into in a cauldron of Prolog, spiced with Lisp.
So won’t it be great to have the simplicity of low-code visual flow based programming and the performance (and concurrency) of Erlang?
[*] = amazing in the sense of extendability (Nodes can easily be added and developed by third parties), understandability (Node-RED terminology is straightforward) and usability (once a rectangle becomes a unit of computation and a line becomes a pathway for data flow)
My development process is best described as flow driven development based around a set of test flows to ensure that node functionality is implemented correctly – meaning that it matches the existing Node-RED functionality.
Test flows are mirrored in a separate repository for better maintainability and also integration with existing Node-RED installations.
The architecture, along with a background to Flow Based Programming and Node-RED has been written.
The codebase as it stands, has many interdependence because of testing flows: assert nodes need to know about web-socket communication, or because of the nature of nodes: the complete node needs to know of completed messages.
While the exception node needs to know about exceptions when they happen.
An architectural diagram of this interconnectedness would be probably be just as confusing as the code itself.
Supported Nodes & Features
This is a non-complete list of nodes that partially or completely work:
| Node | Comment |
|---|---|
| catch | catches exception of selected nodes and of entire flows but not groups |
| change | supports many operators but not all. JSONata in basic form is also supported. |
| complete | is available and can be used on certain nodes, not all |
| debug | only debugs the entire message, individal msg properties aren’t supported. msg count as status is supported. |
| delay | supported static delay not dynamic delay set via msg.delay |
| exec | executing and killing commands is supported but only for commands in spawn mode and set on the node. Appending arguments to commands isn’t supported. Timeouts are supported. Kill messages are also supported. |
| file in | working for files located in /priv |
| function | working for any Erlang. Stop and start also respected. Timeout and more than one output port isn’t supported. |
| http in | working for GET and POST, not available for PUT,DELETE etc |
| http request | basic support for doing rrequests, anything complex probably won’t work |
| http response | working |
| inject | working for most types except for flow, global … |
| join | manual arrays of count X is working, parts isn’t supported |
| json | working |
| junction | working |
| link call | working – dynamic calls also |
| link in | working |
| link out | working |
| markdown | working and supports whatever earmark supports. |
| mqtt in | should be working |
| mqtt out | should be working |
| noop | doing nothing is very much supported |
| split | splitting arrays into individual messages is supported, string, buffers and objects aren’t. |
| status | working |
| switch | most operators work along with basic JSONata expressions |
| template | mustache templating is working but parsing into JSON or YAML isn’t supported |
| trigger | the default settings should work |
Elixir helpers can be added to erlang-red-elixir-helpers repository.
There is nothing stopping anyone from creating a complete node in Elixir provided there is a Erlang “node-wrapper”, i.e., a bit of Erlang code in the src/nodes directory that references the Elixir node.
The initial example markdown node is an Erlang node that references Elixir code. I also wrote an Elixir wrapper function whereby I could have just as easily referenced Earmark directly from the Erlang code. That was a stylist choice.
I intend to use Elixir code for importing Elixir libraries to the project and less coding nodes in Elixir. I simply prefer Erlang syntax. But each to their own 🙂
$ rebar3 get-deps && rebar3 compile
$ rebar3 shell --apps erlang_red
Open the Node-RED visual flow editor in a browser:
$ open -a Firefox http://localhost:9090/node-red
I use docker to develop this so for me, the following works:
prompt$ git clone git@github.com:gorenje/erlang-red.git
prompt$ cd erlang-red
prompt$ docker run -it -v $(pwd)/erlang-red:/code -p 9090:8080 -w /code --rm erlang bash
docker> rebar3 shell --apps erlang_red
Then from the docker host machine, open a browser:
prompt$ open -a Firefox http://localhost:9090/node-red
That should display the Node-RED visual editor.
A release can be bundled together:
$ rebar3 as prod release -n erlang_red
All static frontend code (for the Node-RED flow editor) and the test flow files in priv/testflows are bundled into the release.
Cowboy server will started on port 8080 unless the PORT env variable is set.
A sample Dockerfile Dockerfile.fly is provided to allow for easy launching of an instance as a fly application.
The provided shell script (fly_er.sh) sets some common expected parameters for the launch.
Advanced users may wish to examine the fly launch line therein and adjust for their requirements.
Using the container stack at heroku, deployment becomes a git push heroku after the usual heroku setup:
heroku login–>heroku git:remote -a–>heroku stack:set container–>git push heroku
However the Dockerfile.heroku does not start the flow editor, the image is designed to run a set of flows, in this case (at time of writing) a simple website with a single page.
Basically this flow is the red-erik.org site.
The image does this by setting the following ENV variables:
COMPUTEFLOW=499288ab4007ac6a– flow to be used. This can also be a comma separated list of flows that are all started.DISABLE_FLOWEDITOR=YES– any value will do, if set the flow editor is disabled.
Also be aware that Erlang-RED supports a PORT env variable to specifying the port upon which Cowboy will listen on for connections. The default is 8080.
Heroku uses this to specify the port to connect for a docker image so that its load balancer can get it right.
What the gif shows is executing a simple flow using Erlang as a backend. The flow demonstrates the difference in the switch node of ‘check all’ or ‘stop at first match’.
All nodes are are processes- that is shown on the left in the terminal window.
This example is extremely trivial but it does lay the groundwork for expansion.
To create unit tests for this, Node-RED frontend has been extended with a
“Create Test Case” button on the export dialog:
Test flows are stored in the testflows directory and will be picked up the next time make eunit-test is called. In this way it is possible to create unit tests visually.
Flow tests can also be tested within the flow editor, for more details see below.
The flow test suite is now maintained in a separate repository but is duplicated here.
To better support testing of flows, two new nodes have been created:
“Assert Failed” node cases unit tests to fail if a message reaches it, regardless of any message values. It’s basically the same as a assert(false) call. The intention is to ensure that specific parts of a flow aren’t reached.
The second node (in green) is an equivalent to a change node except it contains test on attributes of the message object. Possible tests include ‘equal’, ‘match’, ‘unset’ and the respective inverses. Here the intention is that a message passes through is tested for specific values else the unit test fails.
These nodes are necessary since there is no other way to test whether flow is working or not.
Also remember these flow tests are designed to ensure the Erlang backend is correctly implementing node functionality. The purpose of these nodes is not to ensure that a flow is correct, rather that the functionality of implemented nodes works and continues to work correctly.
My plan is to create test flows that represent specific NodeRED functionality that needs to be implemented by Erlang-RED. This provides regression testing and todos for the implementation.
I created a keyboard shortcut for creating and storing these test flows from the flow editor directly. However I was still use the terminal to execute tests make eunit-test – which became painful. So instead I pulled this testing into Node-RED, as the gif demonstrates:
What the gif shows is my list of unit tests, which at the press of a button, can all be tested. Notifications for each test shows the result. In addition, the tree list shows which tests failed/succeed (red ‘x’ or green check). Also tests can be executed individually so that failures can be checked individually.
The best bit though is that all errors are pushed to the debug panel and from there I get directly to the node causing the error. Visual unit testing is completely integrated into Erlang-RED.
My intention is to create many small flows that represent functionality that needs to be implemented by Erlang-RED. These unit tests shows the compatibility to Node-RED and less the correctness of the Erlang code.
Contributions very much welcome in the form of Erlang code or as Node-RED test-flows, ideally with the Erlang implementation. Elixir code is also welcome, only it has its own home.
Each test flow should test exactly one feature and use the assert nodes to check correctness of expected results. Tests can also be pending to indicate that the corresponding Erlang functionality is still missing.
Questions and Answers at either the Erlang Forum or the Node-RED Forum.
To branch or not to branch, that isn’t really a question. I’m currently working directly on main but ensure that all tests succeed before pushing, so main branch will always work. Locally I work with branches but have no desire to make those branches public since I’m working on my own.
Versioning is completely random and has little or no meaning at the moment. I prefer to use Milestones since these are arrived at and are not planned, I don’t know when the next milestone will be reached.
If this project becomes more collaborative or a “production ready piece of software”, more certainty will be applied to the development process, i.e., semantic version numbers will introduced.
I’m more than happy to deal with conflicts if someone developed something on a branch and it doesn’t merge – I understand that multiple direct pushes to main everyday isn’t the done thing but I don’t like to have code lying around for weeks on end, not being merged because it’s not on a release schedule.
Coding is a creative process, creativity cannot be planned. Imagine Van Gogh working according to a release plan.
Nick and Dave for bring Node-RED to live – amazing quality and flexibility and the entire Node-RED community.
Much thanks to
- @mwmiller for providing a fly server for running a live version of Erlang-RED,
- @joaohf many tips on coding Erlang and structuring an Erlang project, and
- @Maria-12648430 for debugging my initial attempt to create a gen_server for nodes.
No Artificial Intelligence was harmed in the creation of this codebase. This codebase is old skool search engine (ddg), stackoverflow, blog posts and RTFM technology.
Also be aware that this project partly uses the Don’t do Evil un-enforceable license. The point of the license is not to be enforceable but to make the reader think about what is evil. After all, Pope Leo (the new one) did say “evil will not prevail” – what does that even mean?
