Reverse engineering the Controllino Helium Hotspot

Disclaimer

This document does not contain any source code, copies of scripts or deployment keys, which are all property of Controllino GmbH. If you want to obtain those, you have to buy a Hotspot and open it yourself. This document will however guide you about what to expect and maybe help you make a decision if you want to buy a Controllino Hotspot and maybe splurge for an Apps license.

I've only worked with their original V1 Hotspot so far. Their new Pro version might offer additional features or bugfixes.

Also since the process of opening the device leaves quite a mark on the case, doing so will certainly void any warranty.

With this disclaimer out of the way, let's start.

Why open the device?

That's a good question. There's the old saying: If it ain't broken, don't fix it. But it was broken. Our first hotspot simply wan't accessible anymore, not doing anything. Since there is no way to manage the device except for the not very detailled status page, I decided to open it. Inside there's a Raspberry Pi 4 with 8GB of RAM, a 32GB High Endurance Micro-SD card and a LoraWAN concentrator. Luckily it fixed itself because at the time I started poking around everything was working just fine again.

For another project we wanted to use an LTE modem and be able to manage the device via OpenVPN. LTE modems should work with their new Hotspot Pro, but I doubt Controllino will add OpenVPN capabilities.

And on yet another Hotspot it actually was broken and Controllino was unable to fix it (see below).

How to open the device?

There are bolts on both sides, screwed into threaded spacers inside which hold the whole thing together. One side has a hex insets, the other side uses slot screws. If that were all it would be really easy. The problem is that all the bolts are glued in place. Unscrewing them requires lots of force, often strips the slots and breaks all bolts in half. On one hotspot the slot headed screws were glued down so much I had to drill it open. If you start with the hex bolts and feel confident "Hey, they open up really easy", just wait until you get to the other side.

The Pro version seems to be riveted instead. So there's no way around drilling unless someone comes up with a way of jailbreaking it through the exposed connectors. Those are: One USB 2 port, one USB 3 port and maybe parts of the 40 Pin GPIO header through the ventilation holes.

How to jailbreak the device?

Since I didn't know how Controllino manages their devices, I wanted to be the least invasive as possible. This means changing passwords was out of the question. I figured it would be best to simply add my SSH keys to ~/.ssh/authorized_keys on both accounts and enable SSH. The easiest way to to this is to add "/etc/init.d/ssh start" somewhere in /etc/rc.local. It's also possible to chroot into the system and enabling the systemd service but this requires either using an aarch64 host or qemu userspace emulation.

When in there, it might be a good idea to add some heatsinks onto that Raspberry PIs CPU, as it gets quite toasty - and to make a backup of the SD card in case anything goes wrong.

That's it. Now it can be put back together. I used duct tape since all the bolts were broken. From now on the Raspberry can be accessed via SSH.

How does the software work?

The software is split into three parts:

The host system: A standard Raspberry OS 64 Bit. There are some shell scripts on the host to help with management of the docker containers and system updates. Also the Semtech packet forwarder runs on the host, talks to the LoraWAN concentrator via SPI and to the miner via UDP. Cronjobs are set up to run monitoring and update scripts. One management utility which accepts commands from the control interface over a REST-like API appears to be a compiled Python application (using pyinstall). It should be possible to decompile it but so far I wasn't too interested in it.

The miner: A docker container containing the actual hotspot. This is a standard Docker image provided by Helium, with some configuration parameters passed in. The container is based on Alpine Linux. Access to the Raspberries I2C bus is passed to the container, as the secure element which holds the miners keys are hooked up to it.

The control interface: Another Docker container. Contains a node.js application, developed by Controllino GmbH. This provides the "management web interface". The code is not obfuscated and pulled straight from their Github repository. Quite an interesting read. This container is also based on Alpine Linux.

If broken, what can be repaired?

There was one Hotspot which simply wouldn't sync the blockchain anymore. It got stuck around 20000 blocks and went very very slowly, so that even if left alone it would never catch up. Controllinos advice was to clear out miner data, perform an Instasync and various other things. This hat no effect. Clearing the data would only reset the counter and have it crawl up to 20000 again. Poking around in the system I noticed that the Shell-Script which was supposed to perform the aforementioned Instasync was completely missing. But what even is an Instasync? Helium is a peer-to-peer network, and every node needs to have a copy of the blockchain. Now that Light Hotspots should be enabled, I'm not sure why Controllino Hotspots still need to download the blockchain, but I will investigate this further in the future. Downloading this many blocks from other hotspots and validators using the peer-to-peer mechanism can be quite time consuming so it's possible to download copies of the blockchain from the web and prime the miner with it. Which is exactly what the Instasync script I copied over from another Hotspot is supposed to do.

Actually, before I noticed the missing script, I have already manually performed the actions an Instasync does by downloading the file, unpacking it, placing it into the Docker container and importing it with the "miner" command.

Another problem that can occur is that the Miner Docker container simply stops running and has to be started again manually. Controllino has a monitoring script for their management interface in place, but not for the Miner.

Luckily, the miner can be restarted using the webinterface, so if this is the only problem, opening the device is not necessary.

How do Hotspot Apps work?

Controllino released a new feature and deployed it to all hotspots which allows you to install additional "apps" onto the device. The reasoning is that the hardware is quite powerful and you can have it do more than just mine Helium. It could be the center of your home automation system, mine some VPN based coins or plot out your Crypto earnings.

This, of course, is not free. As I am writing this, an "Apps license" costs 90 Euros per Hotspot. It's non-transferable but once activated you can install as many apps as you like - from their very limited selection, that is.

Of course you could just open the Hotspot and do anything you like with the Raspberry inside. But if you prefer a simple installation process, actually have use for apps they are selling and don't want to void the warranty, then it might be for you.

So what does the app installer do? Previously I mentioned that their management stack is either unobfuscated JavaScript or Shell Script. Ha - but of course not the license checker and app installer. They are contained in a binary written in C. The binary is linked against Alpines variant of libc which is different to standard glibc. This makes reverse engineering a bit harder but far from impossible.

Basically it contains three functions, which can be selected from by passing a parameter on the commandline. The first one activates a license, it does this by talking to the Woocommerce plugin of their Wordpress page. The second one validates an already activated license, again by contacting Woocommerce. The third function performs the actual installation of the software. Of course it validates the license first, calling the second function before doing anything.

I tested this by patching the binary using a Hex-Editor so that the first and second function mentioned above would simply always succeed without checking anything. This gave me access to the apps.

Ok, so how does the installation process look? Well it fetches Docker Images, straight from the developers of the software you want to install - and installs it. There are no modifications done by Controllino GmbH, they are litararily just fetching Docker images and starting them. There is no integration into their management interface (except telling you it's installed and allowing you to uninstall it). Apps are hosted on separate TCP ports. There is no possibility to edit configuration files, do backups or anything.

I really wasn't impressed. Controllino, you could do better.

What else to do?

Add an LTE modem: If you have a modem which emulates an Ethernet port via USB (RNDIS, Huawei calls it HiLink) this is as straight forward as it gets. Just plug it in. Done.

Configure OpenVPN: This allows you to remotely manage your Hotspot when it's within a network you don't have direct access to or hooked up via LTE. We only had issues with the system randomly removing IPv6 connectivity. It might be Dockers fault but I'm not certain. To work around that issue I had to force OpenVPN to connect to our VPN server via IPv4.

Battery monitoring: When adding a UPS or solar charger it might be a good idea to keep an eye on the battery health and level.

Put in a bigger SD card: The SD card keeps getting quite full. It's a good thing that their management scripts keep a close eye on that, because otherwise the system would likely just crash. Whenever the card gets too full the miner data is deleted an an Instasync performed. This will most likely not be an issue anymore with the Light Hotspots, but until then you could increase the storage capacity with a bigger SD-card to delay this process as much as possible. It might also save some bandwidth.

Add PoE: Altough external PoE-Extractors exist, utilizing the PoE capabilities of the Pi4 could make a cheaper or more versatile solution. For example on one non-Helium-related project, I had to power a few Raspberry Pis via PoE. The easiest solution was to hook up 5V DC/DC Regulators to the Raspberry PIs 4 pin Ethernet PoE tap, and feeding the regulated 5V through the GPIO header. The power was coming from passive Unifi 24V Injectors left over from another project.

Use WiFi: Another thing that the Pro version is supposed to provide. But if you remove the Raspberry from the fully enclosed metal casing, getting a good WiFi connection should be possible even with the V1 model.

Remove the Controllino software: So far we haven't done this but it's entirely possible to remove the management scripts and webinterface and manage the Hotspot from the command line. The only reason why you would want this is if you made a lot of modifications and are worried about an automatic update messing them up. Or if you want to experiment with the mining software itself. Although not being very powerful, the management interface still fulfills its functions of onboarding the Miner and giving a quick status update quite well.