Node-RED Flows that make flows

Executive Summary

When a mummy function node and a daddy function node love each other very much, the daddy node injects a payload into the mummy node through a wire between them, and a new baby message flow is made, which comes out of mummy’s output terminal. The baby’s name is Jason.

Where did this idea come from?

In Node-RED, if you export some nodes to the clipboard, you will see the flow and its connections use a JSON wire format.

For a few weeks I’ve been thinking about generating that JSON programmatically, in order to create flows which encode various input and output capabilities and functions, according to some input parameters which give the specific details of what the flow is to do.

I think there’s been an alignment of planets that’s occurred, which led to this idea:

@stef_k_uk has been talking to me about deploying applications onto “slave” Raspberry Pis from a master Pi.

Dritan Kaleshi and Terence Song from Bristol University have been discussing using the Node-RED GUI as a configurator tool for a system we’re working on together, and also we’ve been working on representing HyperCat catalogues in Node-RED, as part of our Technology Strategy Board Internet of Things project, IoT-Bay.

@rocketengines and I were cooking up cool ideas for cross-compiling Node-RED flows into other languages, a while back, and

in a very productive ideas afternoon last week, @profechem and I stumbled upon an idea for carrying the parser for a data stream along with the data itself, as part of its meta-data description.

All of the above led me to think “wouldn’t it be cool if you could write a flow in Node-RED which produced as output another flow.” And for this week’s #ThinkFriday afternoon, I gave it a try.

First experiment

The first experiment was a flow which had an injector node feeding a JSON object of parameters into a flow which generated the JSON for an MQTT input node, a function node to process the incoming data in some way, and an MQTT output node. So it was the classic subscribe – process – publish pattern which occurs so often in our everday lives (if you live the kind of life that I do!).
first Node-RED flowAnd here’s the Node-RED flow which generates that: flow-generator

So if you sent in
{
"broker": "localhost",
"input": "source_topic",
"output": "destination_topic",
"process": "msg.payload = \"* \"+msg.payload+\" *\";\nreturn msg;"
}

the resulting JSON would be the wire format for a three-node flow which has an MQTT input node subscribed to “source_topic” on the broker on “localhost”, a function node which applies a transformation to the data: in this case, wrapping it with an asterisk at each end, and finally an MQTT publish node sending it to “destination_topic” on “localhost”.
N.B. make sure you escape any double quotes in the “process” string, as shown above.

The JSON appears in the debug window. If you highlight it, right-mouse – Copy, and then do Import from… Clipboard in Node-RED and ctrl-V the JSON into the text box and click OK, you get the three node flow described above ready to park on your Node-RED worksheet and then Deploy.
the resulting And it works!!

So what?

So far so cool. But what can we do with it?

The next insight was that the configuration message (supplied by the injector) could come from somewhere else. An MQTT topic, for example. So now we have the ability for a data stream to be accompanied not only by meta-data describing what it is, but also have the code which parses it.
flow with added MQTT configuratorMy thinking is that if you subscribe to a data topic, say:
andy/house/kitchen/temperature
there could be two additional topics, published “retained” so you get them when you first subscribe, and then any updates thereafter:

A metadata topic which describes, in human and/or machine readable form, what the data is about, for example:
andy/house/kitchen/temperature/meta with content
“temperature in degrees Celcius in the kitchen at Andy’s house”

And a parser topic which contains the code which enables the data to be parsed:
andy/house/kitchen/temperature/parser with content
msg.value = Math.round(msg.payload) + \" C\"; return msg;
(that’s probably a rubbish example of a useful parser, but it’s just for illustration!)

If you’re storing your data in a HyperCat metadata catalogue (and you should think about doing so if you’re not – see @pilgrimbeart‘s excellent HyperCat in 15 minutes presentation), then the catalogue entry for the data point could include the URI of the parser function along with the other meta-data.

And then…

Now things get really interesting… what if we could deploy that flow we’ve created to a node.js run-time and set it running, as if we’d created the flow by hand in the Node-RED GUI and clicked “Deploy”?
Well we can!

When you click Deploy, the Node-RED GUI does an HTTP POST to “/flows” in the node.js run-time that’s running the Node-RED server (red.js), and sends it the list of JSON objects which describe the flow that you’ve made.
So… if we hang an HTTP request node off the end of the flow which generates the JSON for our little flow, then it should look like a Deploy from a GUI.

Et voila!
flow that deploys to a remote Node-RED
Note that you have to be careful not to nuke your flow-generating-flow by posting to your own Node-RED run-time! I am posting the JSON to Node-RED on my nearby Raspberry Pi. When you publish a configuration message to the configuration topic of this flow, the appropriate set of nodes is created – input – process – output – and then deployed to Node-RED on the Pi, which dutifully starts running the flow, subscribing to the specified topic, transforming the data according to the prescribed processing function, and publishing it to the specified output topic.

I have to say, I think this is all rather exciting!

@andysc

Footnote:

It’s worth mentioning, that Node-RED generates unique IDs for nodes that look like “8cf45583.109bf8″. I’m not sure how it does that, so I went for a simple monotonically increasing number instead (1, 2 …). It seems to work fine, but there might be good reasons (which I’m sure @knolleary will tell me about) why I should do it the “proper” way.

MQTT powered video wall

Scaling things up a little from my first eightbar post.

This was one of those projects that just sort of “turned up”. About 3 weeks ago one of the managers for the ETS department in Hursley got a call from the team building the new IBM Forum in IBM South Bank. IBM Forums are locations where IBM can showcase technologies and solutions for customers. The team were looking for a way to control a video wall and a projector to make them show specific videos on request. The requests will come from pedestals known as “provokers”, each having a perspex dome holding a thought-provoking item. The initial suggestions had been incredibility expensive and we were asked if we could come up with a solution.

Provoker

The provokers have access to power and an Ethernet connection. Taking all that into account a few ideas came to mind but the best seamed to be an Arduino board with Ethernet support and a button/sensor to trigger the video. There is a relatively new arduino board available that has a built in Ethernet shield which seemed perfect for this project. Also, since a number of the items in the provokers would be related to IBM’s Smarter Planet initiative, it made sense to use MQTT as a messaging layer as this has been used to implement a number of solutions in this space.

Nick O’Leary was enlisted to put together the hardware and also the sketch for the Arduino as he had already written a MQTT client for Arduino in the past.

Each provoker will publish a message containing a playload of “play” to a topic like

provoker/{n}/action

Where ‘{n}’ is the unique number identifying which of the 6 provokers sent the message.

To provide some feedback to the guest that pressed the button, the LED has been made to pulse while one of the provoker-specific videos is playing. This is controlled by each provoker subscribing to the following topic

provoker/{n}/ack

Sending “play” to this topic causes the LED pluse, sending “stop” turns the LED solid again.

The video wall will be driven by software called Scala InfoChannel which has a scripting interface supporting (among other things) Python. So a short script to subscribe to the ‘action’ topics and to publish on on the ‘ack’ got the videos changing on demand.

And sat in the middle is an instance of the Really Small Message Broker to tie everything together.

Arduino in a box

This was also the perfect place to use some of my new “MQTT Inside” stickers.

First sticker deployed

This project only used one of the digital channels (for the button) and one of the analogue channels (for the LED) available on the Arduino – which leaves a lot of room for expansion for these type of devices. I can see them being used for future projects.

Parts list

  1. Arduino Ethernet
  2. Blue LED Illuminated Button
  3. A single resistor to protect the LED
  4. 9v power supply
  5. Sparkfun Case

A different kind of TV remote control

I’m very excited to be welcoming another Hursley innovator as a guest here on eightbar – Benjamin Hardill (you can find him on Twitter as @hardillb). Here’s some insight into what he’s been up to lately! More home automation, hardware hacking, and MQTT messaging adventures follow :-)

I got a new TV around Christmas last year and while unpacking it I noticed along with the HDMI, SCART and other sockets on the back it had a 9-pin socket labelled "RS232C IN CONTROL&SERVICE". I didn’t think that much of it at the time, but I remembered it last week while thinking about a couple of problems that had come up.

Tidy TV setup The first of these was that I had got home twice recently to find I’d left the TV turned on while I was at work, this was mainly because I use MythTV and I’d left it at the menu screen rather than turning the screen off as well. This had left shadow on the menu on the screen for a day or so afterwards (luckily no permanent damage as would have happened with a plasma or CRT TV).

The other point was from when we all first got hold of our Current Cost meters, there had been a lot of thought about how to work out exactly what appliances were on at any given time. While spotting when things like an electric water heater turned on was relatively easy, it was proving difficult to spot some of the lower power devices.

A plan started to form and I ordered a null modem cable from Amazon (£2.18 with free shipping) and went looking for some documentation on the protocol. The manual that came with the TV while being nearly an inch thick just covers the basics of how to plug it in and turn it on, but there was a CD-ROM with a much more detailed PDF document. The version for my TV is here. While searching round I found manuals for several other LG LCD/plasma TVs and they all seem to use the same basic protocol.

The protocol is relatively simple

[cmd1][cmd2] [setid] [data]

Where the cmd1 & cmd2 are 1 letter code, setid is for if you have multiple TVs connected to the same cable, the default id is 01 but you can change if needed, using 00 will work for all connected TVs. And data is a hex value of the option to pass the command.

The response from the TV looks like this for a success

[cmd2] [setid] OK[data]x

and like this for a failure

[cmd2] [setid] NG[data]x

The command to turn the TV on and off is "ka" so sending

ka 00 1

turns the TV on and sending

ka 00 0

turns it off. Most of the commands will reply with the current status if they are passed ff as the data. So sending

ka 00 ff

gets the following when the TV is off

a 00 OK0x

So now I had a way to turn the TV on and off along with checking its current status. The next step was to surface this some way and given the fascination we all seem to have with messaging, MQTT seemed like a good idea. A little bit of Java and the Java COMM API later and I had 2 topics TV/Commands & TV/Status.

I already have a topic that publishes if my mobile phone is in the flat by pinging it with Bluetooth. Combining this with the two new topics I can ensure that the TV is turned off when I leave. I’m also wondering if I should start to log the amount of time the TV is on, but I think the results may scare me a little…

Mad scientists?

IMG_3669 4.JPG IMG_3670 4.JPG

Just an average day in the Hursley tea bar!

Martin Dix from CurrentCost visited to deliver a bunch of meters for people who had attended Home Camp 08. Whilst he was with us, Andy Stanford-Clark hooked up a (battery-powered) meter to one of his mousetraps to show how it is possible to detect traps going off using the meter – this is the same technique he uses in his automated home system. He also showed the same message arriving in his MQTT broker on his Ubuntu laptop. The table was soon awash with gadgets, wires and tools. I brought along a camera or two, Dale brought along some questions about the software internals of the new devices, and a great time was had by a bunch of geeks over tea…

No mice were harmed during this meetup, but a few wooden coffee stirrers were put through their paces in some vicious mousetraps.

Just thinking out loud – Metaverse snapshot

I moved offices today and having a bright new whiteboard I could not leave it clean for long.
Its not really a mindmap, just some association of thoughts and bits of linkages. I am sure it will alter, but right now this is what was in my head in a mad flurry. The underlying red part is really the substrate of the whole thing. Just my personal thoughts linked to some of the things I have seen and been involved with one way and another.

Thoughts on the metaverse
Note: edited to show smaller version of the board as it was cropping the right hand important side for those that did not click through to flickr. 3d printing FTW and high value professional social networks one there too !

Rivers Run Red’s Retail Planogram in Second Life

Thanks again to Malburns for spotting this and tweeting it. Rivers Run Red have released an example of an application layered onto immersive workspaces in Second Life. In this case it is around retail planning and visualization.

This is an example of the next layer of toolsets that we can expect to see across virtual worlds, as those virtual worlds become a platform not just a place.
Producing what if scenarios, or mirror world scenarios does need the ability to simply sketch and examine the possibilities whether its a retail store, a machine room or an intricate business model that cannot normally be visualized.
The exciting thing about this for us here at eightbar is that it makes it a step closer to be able to then instrument the model with real live data via publish subscribe methods such as MQTT. Merging the data from a smart planet into immersive visualizations that can be explored together, not stand alone clearly is a direction we have been pushing since even before the 2006 Wimbledon. Hursley is (for those who dont know) the home of messaging, pub/sub reliable MQ messaging.