Node-RED Flows that make flows

Executive Summary

When a mummy function node and a daddy function node love each other very much, the daddy node injects a payload into the mummy node through a wire between them, and a new baby message flow is made, which comes out of mummy’s output terminal. The baby’s name is Jason.

Where did this idea come from?

In Node-RED, if you export some nodes to the clipboard, you will see the flow and its connections use a JSON wire format.

For a few weeks I’ve been thinking about generating that JSON programmatically, in order to create flows which encode various input and output capabilities and functions, according to some input parameters which give the specific details of what the flow is to do.

I think there’s been an alignment of planets that’s occurred, which led to this idea:

@stef_k_uk has been talking to me about deploying applications onto “slave” Raspberry Pis from a master Pi.

Dritan Kaleshi and Terence Song from Bristol University have been discussing using the Node-RED GUI as a configurator tool for a system we’re working on together, and also we’ve been working on representing HyperCat catalogues in Node-RED, as part of our Technology Strategy Board Internet of Things project, IoT-Bay.

@rocketengines and I were cooking up cool ideas for cross-compiling Node-RED flows into other languages, a while back, and

in a very productive ideas afternoon last week, @profechem and I stumbled upon an idea for carrying the parser for a data stream along with the data itself, as part of its meta-data description.

All of the above led me to think “wouldn’t it be cool if you could write a flow in Node-RED which produced as output another flow.” And for this week’s #ThinkFriday afternoon, I gave it a try.

First experiment

The first experiment was a flow which had an injector node feeding a JSON object of parameters into a flow which generated the JSON for an MQTT input node, a function node to process the incoming data in some way, and an MQTT output node. So it was the classic subscribe – process – publish pattern which occurs so often in our everday lives (if you live the kind of life that I do!).
first Node-RED flowAnd here’s the Node-RED flow which generates that: flow-generator

So if you sent in
{
"broker": "localhost",
"input": "source_topic",
"output": "destination_topic",
"process": "msg.payload = \"* \"+msg.payload+\" *\";\nreturn msg;"
}

the resulting JSON would be the wire format for a three-node flow which has an MQTT input node subscribed to “source_topic” on the broker on “localhost”, a function node which applies a transformation to the data: in this case, wrapping it with an asterisk at each end, and finally an MQTT publish node sending it to “destination_topic” on “localhost”.
N.B. make sure you escape any double quotes in the “process” string, as shown above.

The JSON appears in the debug window. If you highlight it, right-mouse – Copy, and then do Import from… Clipboard in Node-RED and ctrl-V the JSON into the text box and click OK, you get the three node flow described above ready to park on your Node-RED worksheet and then Deploy.
the resulting And it works!!

So what?

So far so cool. But what can we do with it?

The next insight was that the configuration message (supplied by the injector) could come from somewhere else. An MQTT topic, for example. So now we have the ability for a data stream to be accompanied not only by meta-data describing what it is, but also have the code which parses it.
flow with added MQTT configuratorMy thinking is that if you subscribe to a data topic, say:
andy/house/kitchen/temperature
there could be two additional topics, published “retained” so you get them when you first subscribe, and then any updates thereafter:

A metadata topic which describes, in human and/or machine readable form, what the data is about, for example:
andy/house/kitchen/temperature/meta with content
“temperature in degrees Celcius in the kitchen at Andy’s house”

And a parser topic which contains the code which enables the data to be parsed:
andy/house/kitchen/temperature/parser with content
msg.value = Math.round(msg.payload) + \" C\"; return msg;
(that’s probably a rubbish example of a useful parser, but it’s just for illustration!)

If you’re storing your data in a HyperCat metadata catalogue (and you should think about doing so if you’re not – see @pilgrimbeart‘s excellent HyperCat in 15 minutes presentation), then the catalogue entry for the data point could include the URI of the parser function along with the other meta-data.

And then…

Now things get really interesting… what if we could deploy that flow we’ve created to a node.js run-time and set it running, as if we’d created the flow by hand in the Node-RED GUI and clicked “Deploy”?
Well we can!

When you click Deploy, the Node-RED GUI does an HTTP POST to “/flows” in the node.js run-time that’s running the Node-RED server (red.js), and sends it the list of JSON objects which describe the flow that you’ve made.
So… if we hang an HTTP request node off the end of the flow which generates the JSON for our little flow, then it should look like a Deploy from a GUI.

Et voila!
flow that deploys to a remote Node-RED
Note that you have to be careful not to nuke your flow-generating-flow by posting to your own Node-RED run-time! I am posting the JSON to Node-RED on my nearby Raspberry Pi. When you publish a configuration message to the configuration topic of this flow, the appropriate set of nodes is created – input – process – output – and then deployed to Node-RED on the Pi, which dutifully starts running the flow, subscribing to the specified topic, transforming the data according to the prescribed processing function, and publishing it to the specified output topic.

I have to say, I think this is all rather exciting!

@andysc

Footnote:

It’s worth mentioning, that Node-RED generates unique IDs for nodes that look like “8cf45583.109bf8″. I’m not sure how it does that, so I went for a simple monotonically increasing number instead (1, 2 …). It seems to work fine, but there might be good reasons (which I’m sure @knolleary will tell me about) why I should do it the “proper” way.

Homecamp 4

DC power, electricity monitoring, gas monitoring, data gathering, solar panels, hardware hacking, software hacking, behaviour change, open source, home energy, energy visualisations, cows…

Just some of the things we talked about this weekend at Homecamp 4 at the Centre for Creative Collaboration near King’s Cross in London.

A two-day event this year, Homecamp 4 (thanks to organiser Ken Boak) brought together hardware hackers, software hackers, home automation fans, energy and sustainability people, designers, and loads of enthusiasm and energy for sharing skills and experiences. And, thanks to the generous sponsorship by Amee and the support of Manager Debbie Davies at C4CC, we had lovely breakfasts, lunches, and Saturday evening drinks.

C4CC

Saturday morning was dedicated to a series of presentations that had been scheduled in advance. It kicked off with a fascinating presentation by Simon Daniel from Moixa (inventor of the Palm folding keyboard and the USB rechargeable batteries) about how we could, and should, power all our lighting and computing at home off a couple of solar panels. By storing the solar power in batteries and then using it as DC power, rather than converting it to AC power, we’d not only avoid unsightly power bricks on everything, but we’d double the efficiency of the solar energy over converting it to the usual AC power to match our ordinary home electricity circuits and then convert it back to DC for consumption by LED lighting, laptops, mobile phones, and the like.

Simon from Moixa

Another interesting point he made was that most young people either don’t own their own homes or they’re not likely to stay in their current house for the next 15 years so they’re unlikely to invest in solar panels. So the company is working out a way to produce essentially portable solar panel, battery, and control packages that could be installed (and transported when you move house) as easily as Sky.

Next up was James Smith from Amee to demo what you can do with the Amee API. Amee collect data about energy consumption and carbon emissions, most of which is publicly available but usually not in a form that’s easy to consume. A team of scientists search out data from all over the world about the carbon emissions of anything you can think of. The team reviews the data, corrects any errors they find, and then turns the data into code. Developers can then write applications that use the data through the Amee API.

James from Amee

James and Chris Adams, also from Amee, demo’d how to use one of the toolkits that Amee provides to developers to show how quick it is to generate a web form that calculates the carbon emissions of cows from ‘enteric fermentation’ (burping and farting, to you and me). His favourite Amee hack, though, was to display the carbon emissions of setting fire to things in Minecraft.

After a brief pitch by Casper Koomen from Amsterdam about Pachube, and an update from organiser Ken on his Nanode (previously seen, discussed, and built at Oggcamp 11!), Trystan Lea and Glyn Hudson (from North Wales) described and demo’d their neat open source, Arduino-based energy monitoring project, the OpenEnergyMonitor (first presented at Homecamp 2).

After a novel but tasty free lunch of Vietnamese baguettes, the afternoon proceeded in a more unconference-style format. In a mix of formal presentations and informal discussions, we had talks on a range of topics, including getting your hardware ideas produced commercially, home energy monitoring experiences and visualisations, home-grown wind turbines, 3D printing, and I gave a short introduction/overview on the psychology of energy behaviour change. Most talks got several questions and I was really pleased that my talk, despite having a fairly different focus from many of the others, generated quite a lot of interest and discussion both at the time and over the rest of the weekend.

Evidence I was there!

The Saturday evening pub was unfortunately closed for a private party so Tony and I encouraged everyone back to the lovely Harrison Bar where we were staying and had already eaten a good dinner and noticed an interesting list of beers. If you like ginger beer, I recommend the Crabbie’s Ginger Beer (4%).

Sunday took on a more hackday feel and the attendees either hacked on Nanodes and similar hardware, played with software APIs, or wandered/sat around discussing cool ideas.

Hacking

In all it was a very fun weekend. We had some great conversations and met some very cool people. Hopefully, there’ll be a Homecamp 5.

Conversing

During the weekend, Tony and I, ever on the lookout for interesting content for the Ubuntu-UK Podcast, interviewed James about Amee, and Tristan and Glynn about their OpenEnergyMonitor project. Look out for them in upcoming episodes of UUPC.

Photos thanks to Tony Whitmore

Minihacks and Open Technologies

It’s not all about process, software development, and quadricopters… :-)

Guruplug This week we’ve had what could be described as a “mini Hackday”, instigated by an idea from Andy Stanford-Clark and organised by Hursley newcomer Vaibhavi Joshi. The idea was to spend a few hours exploring the world of plug computers (in this case, a model called a Guruplug); to brainstorm some ideas around utility computers; and to generally see what we could do with this kind of a form factor.

Some great ideas emerged, and quite a few of us were severely tempted to order our new shiny gadgets on the spot… by the end of the morning the Really Small Message Broker was built and running on the Guruplug and some exciting MQTT-related thoughts were flying around. A nice break from the norm for all of us!

Inspired by some of the “social technologies for internal communications” discussions I’d had with Abi Signorelli at Social Media Week London the previous week – in particular, the ease of capturing a brief audio snippet on any particular topic – I thought I’d ask Vaibhavi what she thought – here’s a quick interview:

Straight after the hacking, it was time to move on to the Open Technologies event that was being run to promote Linux, Firefox and Symphony. I’m a user and a big fan of all of these tools so it was nice to see a local Hursley event as part of IBM’s global awareness month dedicated to helping those within the internal community not yet up-to-speed on what people were using. The best part? Free stickers :-)

Open Technologies