MVP Dashboard in NodeRed


#1

I was playing around with Node-Red and the existing OpenAG-MVP-II Software to see what it would look like to have everything on one page. Currently it uses the MVP/logs/cron.log to grab it’s data although you could run the python/bash scripts directly from Node-Red. The buttons and the form don’t call anything yet, but in the future:

  • Configuration: Place to configure modules, GPIO pins, and any other values to use in the bash and python scripts.
  • Temperature/Humidity: Full sized graphs with more history
  • Image: Full sized image and ability to snap a new picture
  • Submit/Cancel: Submit or Cancel form data to couchdb

Anyways the data to try it out in Node-Red is below the image. Let me know what you think or any improvements that can be made. It can be used in parallel with the existing web frontend.

Node-Red Code (Cut the text below - then in Node-RED use Ctrl-I (Import) - Ctrl-V (Paste) to insert)

[{"id":"bff8da71.27d45","type":"tail","z":"a6d104dd.55b698","name":"TAIL of data","filetype":"text","split":true,"filename":"/home/pi/MVP/logs/cron.log","x":130,"y":80,"wires":[["5054baf4.7f3334"]]},{"id":"b8b46f1c.616aa8","type":"debug","z":"a6d104dd.55b698","name":"TailCron","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":660,"y":80,"wires":[]},{"id":"5054baf4.7f3334","type":"json","z":"a6d104dd.55b698","name":"","property":"payload","action":"str","pretty":true,"x":470,"y":80,"wires":[["b8b46f1c.616aa8","6ae4f45c.a630c4"]]},{"id":"630ec7a4.e78e68","type":"comment","z":"a6d104dd.55b698","name":"Tail of Cron log file","info":"","x":130,"y":40,"wires":[]},{"id":"6ae4f45c.a630c4","type":"link out","z":"a6d104dd.55b698","name":"CronLog","links":["c0903cfc.198a1"],"x":655,"y":120,"wires":[]},{"id":"5806e182.8934f8","type":"comment","z":"a6d104dd.55b698","name":"Convert and UI","info":"","x":120,"y":160,"wires":[]},{"id":"c0903cfc.198a1","type":"link in","z":"a6d104dd.55b698","name":"Temperature","links":["6ae4f45c.a630c4","31a34502.dc02f2"],"x":75,"y":240,"wires":[["785bfa6a.7bda1c","fe0498b1.244f8"]]},{"id":"fe0498b1.244f8","type":"debug","z":"a6d104dd.55b698","name":"AllCronData","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":190,"y":200,"wires":[]},{"id":"785bfa6a.7bda1c","type":"switch","z":"a6d104dd.55b698","name":"","property":"payload","propertyType":"msg","rules":[{"t":"cont","v":"Current Temp:","vt":"str"},{"t":"cont","v":"CurrentFanOn:","vt":"str"},{"t":"cont","v":"Success, humidity","vt":"str"},{"t":"cont","v":"Success, light","vt":"str"}],"checkall":"true","repair":false,"outputs":4,"x":170,"y":280,"wires":[["7accafb3.3d1c08"],["e9c283f.33e67"],["f3c174e2.c6ddb8"],["2b409e3c.ed9eba"]]},{"id":"32a459f2.e19426","type":"debug","z":"a6d104dd.55b698","name":"Temp","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":530,"y":160,"wires":[]},{"id":"27290314.bc660c","type":"ui_gauge","z":"a6d104dd.55b698","name":"Temperature","group":"2c0df990.86a1de","order":1,"width":"3","height":"3","gtype":"gage","title":"Temperature","label":"C","format":"{{msg.payload}}","min":"10","max":"30","colors":["#00b500","#e6e600","#ca3838"],"seg1":"23","seg2":"25","x":690,"y":180,"wires":[]},{"id":"63d8ca34.1ba4dc","type":"ui_chart","z":"a6d104dd.55b698","name":"","group":"8fb3c9cb.472ce","order":1,"width":"6","height":"5","label":"Temperature","chartType":"line","legend":"false","xformat":"dd HH:mm","interpolate":"linear","nodata":"","dot":false,"ymin":"15","ymax":"27","removeOlder":"1","removeOlderPoints":"1000","removeOlderUnit":"86400","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":690,"y":220,"wires":[[],[]]},{"id":"9bc8f5d0.6e362","type":"debug","z":"a6d104dd.55b698","name":"Fan","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":530,"y":260,"wires":[]},{"id":"c8346ca6.fa68a8","type":"debug","z":"a6d104dd.55b698","name":"Humid","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":530,"y":320,"wires":[]},{"id":"e9c283f.33e67","type":"function","z":"a6d104dd.55b698","name":"","func":"var fan = msg.payload.split(':')[1].trim();\nvar result = \"Off\"; //false;\n\nif(fan === \"True\") {\n result = \"On\"; //true;\n}\n\nvar newMsg = { \n payload: result \n};\n\nreturn newMsg","outputs":1,"noerr":0,"x":390,"y":280,"wires":[["9bc8f5d0.6e362","87f8ece4.18cbf"]]},{"id":"7accafb3.3d1c08","type":"function","z":"a6d104dd.55b698","name":"","func":"var temperature = Number(msg.payload.split(':')[1]);\n\nvar newMsg = { \n payload: temperature.toFixed(1)\n};\n\nreturn newMsg;","outputs":1,"noerr":0,"x":390,"y":200,"wires":[["32a459f2.e19426","27290314.bc660c","63d8ca34.1ba4dc"]]},{"id":"87f8ece4.18cbf","type":"ui_text","z":"a6d104dd.55b698","group":"2c0df990.86a1de","order":3,"width":"2","height":"3","name":"","label":"Fan","format":"{{msg.payload}}","layout":"col-center","x":670,"y":280,"wires":[]},{"id":"f3c174e2.c6ddb8","type":"function","z":"a6d104dd.55b698","name":"","func":"//var humidity = msg.payload.split(',');\nvar humidity = Number(msg.payload.split(',')[4]);\n\nvar newMsg = { \n payload: humidity.toFixed(1) //temperature.toFixed(1)\n};\n\nreturn newMsg;","outputs":1,"noerr":0,"x":390,"y":360,"wires":[["c8346ca6.fa68a8","5ce95801.c6cb9","91a31d76.f9eeb"]]},{"id":"91a31d76.f9eeb","type":"ui_chart","z":"a6d104dd.55b698","name":"","group":"8fb3c9cb.472ce","order":2,"width":"6","height":"5","label":"Humidity","chartType":"line","legend":"false","xformat":"dd HH:mm","interpolate":"linear","nodata":"","dot":false,"ymin":"30","ymax":"80","removeOlder":"1","removeOlderPoints":"1000","removeOlderUnit":"86400","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":680,"y":380,"wires":[[],[]]},{"id":"5ce95801.c6cb9","type":"ui_gauge","z":"a6d104dd.55b698","name":"Humidity","group":"2c0df990.86a1de","order":2,"width":"3","height":"3","gtype":"gage","title":"Humidity","label":"%","format":"{{msg.payload}}","min":"30","max":"80","colors":["#00b500","#e6e600","#ca3838"],"seg1":"60","seg2":"70","x":680,"y":340,"wires":[]},{"id":"2b409e3c.ed9eba","type":"function","z":"a6d104dd.55b698","name":"","func":"var light = msg.payload.split(',')[4].trim();\n\nvar newMsg = { \n payload: light\n};\n\nreturn newMsg;","outputs":1,"noerr":0,"x":390,"y":440,"wires":[["c0c00fdc.19b5a","9278a228.42692"]]},{"id":"c0c00fdc.19b5a","type":"ui_text","z":"a6d104dd.55b698","group":"2c0df990.86a1de","order":4,"width":"2","height":"3","name":"","label":"Light","format":"{{msg.payload}}","layout":"col-center","x":670,"y":440,"wires":[]},{"id":"9278a228.42692","type":"debug","z":"a6d104dd.55b698","name":"Light","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":530,"y":400,"wires":[]},{"id":"19a659a6.87a2d6","type":"ui_button","z":"a6d104dd.55b698","name":"","group":"8d5f47d9.57c268","order":1,"width":0,"height":0,"passthru":false,"label":"Configuration","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":170,"y":540,"wires":[[]]},{"id":"db9b6120.e829f","type":"ui_button","z":"a6d104dd.55b698","name":"","group":"8d5f47d9.57c268","order":2,"width":0,"height":0,"passthru":false,"label":"Temperature","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":170,"y":580,"wires":[[]]},{"id":"99b57937.65a1c","type":"ui_button","z":"a6d104dd.55b698","name":"","group":"8d5f47d9.57c268","order":3,"width":0,"height":0,"passthru":false,"label":"Humidity","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":160,"y":620,"wires":[[]]},{"id":"8ad6e887.6c52a","type":"ui_form","z":"a6d104dd.55b698","name":"","label":"","group":"8d5f47d9.57c268","order":8,"width":0,"height":0,"options":[{"label":"Measurement (if exists).","value":"value","type":"text","required":true},{"label":"Notes of significance (optional).","value":"comment","type":"text","required":false}],"formValue":{"value":"","comment":""},"payload":"","topic":"","x":410,"y":660,"wires":[[]]},{"id":"5a4ca01c.b94ba","type":"ui_dropdown","z":"a6d104dd.55b698","name":"","label":"Subject:","place":"Substance of interest.","group":"8d5f47d9.57c268","order":5,"width":0,"height":0,"passthru":true,"options":[{"label":"Plant","value":"plant","type":"str"},{"label":"Reservoir","value":"reservoir","type":"str"}],"payload":"","topic":"","x":420,"y":540,"wires":[[]]},{"id":"86aaf4ff.40705","type":"ui_dropdown","z":"a6d104dd.55b698","name":"","label":"Attribute:","place":"What is being measured.","group":"8d5f47d9.57c268","order":6,"width":0,"height":0,"passthru":true,"options":[{"label":"Planting","value":"plant","type":"str"},{"label":"Germination","value":"reservoir","type":"str"},{"label":"Harvest","value":"harvest","type":"str"},{"label":"Height","value":"height","type":"str"},{"label":"Weight","value":"weight","type":"str"},{"label":"pH","value":"pH","type":"str"},{"label":"Electrical Conductivity","value":"EC","type":"str"}],"payload":"","topic":"","x":420,"y":580,"wires":[[]]},{"id":"60357bd0.0f50c4","type":"ui_dropdown","z":"a6d104dd.55b698","name":"","label":"Status:","place":"End state of the process.","group":"8d5f47d9.57c268","order":7,"width":0,"height":0,"passthru":true,"options":[{"label":"Success","value":"Success","type":"str"},{"label":"Failure","value":"Failure","type":"str"}],"payload":"","topic":"","x":420,"y":620,"wires":[[]]},{"id":"b133bb89.bd5d2","type":"comment","z":"a6d104dd.55b698","name":"Buttons","info":"","x":110,"y":500,"wires":[]},{"id":"9f1d1a56.df691","type":"comment","z":"a6d104dd.55b698","name":"Form","info":"","x":370,"y":500,"wires":[]},{"id":"8ae8456a.04959","type":"ui_button","z":"a6d104dd.55b698","name":"","group":"8d5f47d9.57c268","order":3,"width":0,"height":0,"passthru":false,"label":"Image","color":"","bgcolor":"","icon":"","payload":"","payloadType":"str","topic":"","x":150,"y":660,"wires":[[]]},{"id":"2c0df990.86a1de","type":"ui_group","z":"","name":"Image","tab":"b3e81b6.d3bb668","order":2,"disp":false,"width":"12","collapse":false},{"id":"8fb3c9cb.472ce","type":"ui_group","z":"","name":"Temperature","tab":"b3e81b6.d3bb668","order":3,"disp":false,"width":"6","collapse":false},{"id":"8d5f47d9.57c268","type":"ui_group","z":"","name":"Buttons","tab":"b3e81b6.d3bb668","order":1,"disp":false,"width":"5","collapse":false},{"id":"b3e81b6.d3bb668","type":"ui_tab","z":"","name":"OpenAG MVP","icon":"dashboard"}]


#2

That looks cool. Can you explain more about what you did and how it works? I’ve never heard of node-red before. From a quick glance, their website looks interesting.

@Webb.Peter I’m curious what you think of this.


#3

In short: Node-RED is a flow-based development tool developed originally by IBM for wiring together hardware devices, APIs and online services as part of the Internet of Things. Wikipedia

Basically you drag a widget from the left side and modify the inputs to do what you want. More complex stuff is done through a function written in Javascript but there are modules to use python instead. The code for it is in the previous post (Look for “Node-Red Code” and expand) and can be directly inserted into Node Red.


#4

@wsnook I forgot to mention it in my previous post, generally the UI I created just tail’s the end of the cron log file from the MVP and displays using the node-red-dashboard flow. This tutorial (Node Red - Control RaspberryPi) can give you a quick overview of how to use it.


#5

@webbhm I wonder if this node-red stuff might be useful for you as a way to build user interfaces for charting and control. I know you had concerns in the past about security with CouchDB. Here’s a page on securing node-red.


#6

My first concern is always access. If the MVP lives in a school where we cannot open ports to directly access the Pi from the Web will this still work? In other words, Can the Pi actively send it’s data out to the cloud and query for updates.


#7

@pspeth
With the model of a closed firewall, the CouchDB security is not an issue, all data will be ‘pushed’ out to the cloud somewhere, and the ‘incoming’ will be via a an IoT type of message. CouchDB (or anything else on the Pi) can be open and school’s firewall will give you protection (but not from the students!!). If node-red provides better (easier) charting than writing Python, it is a possible option.
If you can run a browser and query the internet, you can push data via CouchDB replication and and IoT messaging. The problem we are dealing with is setting up a cloud service and finding convenient tools that the community can use.


#8

@rbarta
Just realized that node-red is pre-installed on the Raspberry!! My test Pi is already bloated with MariaDB, CouchDB and all the MVP stuff, and I was afraid of loading a new application of top of all this. Starting up node-red definitely pushes the CPU, but it is running.
With mqtt and http built in, we can easily send messages from an MVP (great for current temperature, etc), or query CouchDB (in the cloud) for charts. I need to look into how the charting works.
When I get the Test MVP stable again, I may need to spin up another Pi to act as a test Cloud service and try running node-red from it.
Any thoughts on how to manage multiple schools/MVPs; where we need separate instances for each?


#9

I’m optimistic that it may be possible to standardize on the API and cloud platform that Rob is working on. From what I’ve seen, it looks like Rob plans to move away from the idea of replicating CouchDB instances and instead use google cloud platform tools.

The basic model I’m imagining is that PFC, food server, etc. devices will have some sort of local GUI or command line tools that make it possible to monitor and control the device at, for example, a school classroom with no reliable internet access. The GUI or command line tools could potentially be different for each type device–perhaps in time we’ll come up with one thing that works well across a range of hardware. Then, there would be another component to push data to, and potentially receive commands from, Rob’s API.

In that scenario, node-red, or something like it, would primarily be useful as a local GUI for use in setup and in-classroom monitoring. You could run it local on the Pi and hit http://localhost from a browser running on the Pi. Or, if you set up some authentication (e.g. maybe different permission levels for teacher & IT department, students in the class, and everybody else), you could run an open port on the local LAN segment and let people do stuff with Chromebooks, iPads, or whatever. It wouldn’t matter that traffic couldn’t get to it from outside the school’s firewall.

The point wouldn’t be off-site access or building a cloud platform–just a nice local GUI that helps to buffer a classroom setup from unpredictable wifi and/or internet connectivity. This would serve as a foundation for reliable operation of the device, but not necessarily as the normal way of interacting with it. In situations where network access is reliable, the cloud platform tools could potentially be the main UI. In that case, the food computer could be configured to regularly poll the cloud API with outbound http connections–probably this would work fine with school firewalls. As you said, if you can browse the web, that ought to work.


#10

@rbarta, @pspeth, @wsnook
I am going to try and spin up Node-Red on a spare Pi and try it as a cloud server. I can add another function to my reporting code to send out a mqtt message of the report data, and see if Red can display the chart in a dashboard. I already have the temperature messaged out on a periodic basis.
At this time, I see Red as a UI tool, and not having it take over the running of the MVP. Likely it could do a lot on the MVP, but what I have running currently is simple and sufficient.


#11

@rbarta
I imported the code and have some missing nodes. I installed the node-red-contrib-ui_j and formio-red, which got some, but still missing ui_form and ui_dropdown. What do I need to install for these? Is there a dashboard behind this, or did you create one from the parts?

Just installed node-red-dashboard, and that took care of it.


#12

@webbhm Yes node-red-dashboard was the one for the UI components, if I remember correctly it’s the only one need for the code. If the scripts would output data to a mqtt topic, then it would allow for the database and UI to either be run on the same Raspberry Pi or be sent to a separate server which would allow for a multiple MVP environment. I briefly looked at the Sensor Data Modeling post which I assume is the structure of the data within the MVP. I have an idea of what might work … I’ll write something up over the next couple of days and you can let me know what you think.


#13

@rbarta
Node-Red may be a good idea, but it is frustrating in practice. Some things are working well, and others are just not (with no explanation). I installed the latest version (0.18.4) and added the dasboard. I created a simple mqtt->JSON->text and while the debug shows the mqtt messagesl; after deploy and opening the UI (localhost:1880/ui), nothing is showing up (tab is “undefined”).

Node-Red Code

[{“id”:“3e05c587.acd76a”,“type”:“tab”,“label”:“MVP-Dashboard”,“disabled”:false,“info”:“Dashboard for the basic MVP”},{“id”:“92115bf1.f5a438”,“type”:“mqtt in”,“z”:“3e05c587.acd76a”,“name”:"",“topic”:“OpenAgBloom/#”,“qos”:“0”,“broker”:“7d1446f2.d3a7d8”,“x”:131,“y”:126,“wires”:[[“cec8c114.44925”]]},{“id”:“fcc2b5a4.83ba08”,“type”:“debug”,“z”:“3e05c587.acd76a”,“name”:"",“active”:true,“tosidebar”:true,“console”:false,“tostatus”:false,“complete”:“false”,“x”:706,“y”:95,“wires”:[]},{“id”:“cec8c114.44925”,“type”:“json”,“z”:“3e05c587.acd76a”,“name”:"",“property”:“payload”,“action”:“str”,“pretty”:false,“x”:340,“y”:126,“wires”:[[“fcc2b5a4.83ba08”,“6486f607.0c1958”]]},{“id”:“6486f607.0c1958”,“type”:“ui_text”,“z”:“3e05c587.acd76a”,“tab”:“23de4b3.cf865b4”,“name”:“Current Temp”,“group”:“9ec63342.4e9b6”,“order”:1,“format”:"{{msg.payload}}",“x”:725,“y”:188,“wires”:[]},{“id”:“7d1446f2.d3a7d8”,“type”:“mqtt-broker”,“z”:"",“name”:“Mosquito”,“broker”:“test.mosquitto.org”,“port”:“1883”,“clientid”:"",“usetls”:false,“compatmode”:false,“keepalive”:“60”,“cleansession”:true,“willTopic”:"",“willQos”:“0”,“willPayload”:"",“birthTopic”:"",“birthQos”:“0”,“birthPayload”:""},{“id”:“23de4b3.cf865b4”,“type”:“ui_tab”,“z”:"",“name”:“MVP Dashboard”,“icon”:“dashboard”,“order”:“1”}]


#14

Unfortunately you need to remove node-red-contrib-ui_j but it is in use (orange)

  1. Delete “Current Temp” (A)
  2. Go to Dashboard and delete “MVP Dashboard” (B, C, D)
  3. You should see under “Manage Palette” that node-red-contrib-ui_j can now be removed
  4. Restart node-red
  5. Add back in the text ui (it will now use the one from node-red-dashboard)
  6. It should now display the temperature.

I would probably delete formio-red as well. The form that I used was part of node-red-dashboard already.


#15

Thanks.
With a Pi, it was easier to just rebuild the system (since this was the only thing on the SD card). It is now up and running. If nothing else, this is a nice, quick way to prototype a UI.


#16

Node Red has some great features, and is well worth using for personal usage; but unfortunately it has some critical limitations which keep it from being viable for the MVP.
The main limitation (at this time) is that Node Red can only support a single UI instance (https://github.com/node-red/node-red-dashboard/issues/263). We would not be able to have a separate UI for each school or box. For a cloud UI, we need the ability to dynamically spin up an interface for each new registered box.
Node Red comes pre-installed on the Raspberry Pi, but it is advised to upgrade to the latest version. I loaded it up on a spare Pi, and ran it independently of the MVP.
I initially had trouble getting Node Red started, I wanted a dashboard, and loaded the libraries for modules (text display, chart) that were not comparable with the dashboard - there were no errors displayed, things just didn’t run. After getting that straightened out (loading the dashboard library loads all the required compatable modules), things went smoothly and simply. It is a simple drag-and-drop graphic building of the system, with filling in some attributes on each module. I created a mqtt client to get data from my Test MVP, then parsed and routed the messages to the right text/gauge/graph. I liked that there is a function module that will run Python (no need to learn JavaScript!!).
i have not tried the text input, but I assume it would be equally easy to build input forms (Phenotype data input, user registration, …) that would meet most of the MVP needs.
Node Red seemed to be a bit intensive on the resources, which is not surprising for such an abstract graphical system. I don’t think it would run well with CouchDB and MySQL all running on the same Pi; but most people are not setting up a test system like I have.
Security should be simple, but following the instructions did not work (I could not locate the package to generate the encrypted password). I am sure could get it working if I spent more time digging into the current documentation, but did not spend the time.
I really got to liking Node Red, and wish we could spin up multiple instances in the cloud (which is suppose to be a feature in the future), but for now it is limited to personal use.


#17

I think it was mentioned earlier that this might work for the Local administration, and use a different solution to view and work with the Cloud. Does this seem viable to you or do you think this would complicated things?


#18

You can always try it. My concern is with how hard this is pushing the Pi. If you have a separate Pi for Node Red, do you end up with one for each MVP? You might have a ‘switch’ by which you tell it which MVP to get messages from, but then the problem is with the charts that cache history. It definitely would have the advantage of keeping all student data (participant) local.
Ultimately, I think we need a cloud solution.


#19

Can you elaborate on how you’re storing/accessing this image? Is it all just from CouchDB?


#20

Nothing pretty. Just calling /home/pi/MVP/scripts/webcam.sh when you click the button which stores it in /home/pi/MVP/pictures/. The picture in the Node-Red GUI is read directly from /home/pi/MVP/web/image.jpg which gets recreated by cron by $MVP_SCRIPTS/render.sh