The Road To Stream Deck Control

November 5th 2022, 10:03 am

Awhile back a friend linked me to a video tour of somebody's home arcade that was posted on YouTube. At one point they showed off that they had a wall-mounted phone with pictures of the games that when tapped would switch the game on. Wow! The idea grabbed me. What a cool addition to a home setup! The way I've been turning my games on and off for over a decade is by flipping a physical switch on a bunch of power strips that each game is plugged into. Its slow, hard on the knees and doesn't look even remotely cool. So I started researching how to set up a slick control scheme similar to the one in the video.

Above all else, the most important goal for my own setup was "no internet of shit!". Basically, no external servers, no required phone apps, no accounts, no monthly subscription fees, no chance of the company going bust in several years like every company in the home automation space seems prone to. I don't want to lose control over my own things next time there's an AWS outage. I don't want to have to re-learn a new ecosystem in three years when the product line I invested in is deemed unprofitable and discontinued. I don't want my usage patterns monitored and data sold so that some company can subsidize what they sell to me. If I couldn't manage this goal, the project would be a non-starter.

I also wanted fast control. In my research I saw a lot of different people talk about home automation setups where it takes 5-10 seconds to turn on a light when tapping a digital switch. That's unacceptably slow. I wanted my stuff to behave much like it did before: instant power-on when a switch was flicked. Finally, if at all possible I wanted everything to be open-source. The space seems so immature still that I didn't want to be locked into weird quirks and caveats of a particular implementation. I wanted something that would be fully customizable to how I imagined it and I didn't want to have to work around firmware bugs or unimplemented features.

After a lot of research I landed on Home Assistant for the brains of the system. It's got a thriving ecosystem with tons of integrations and a super cool way to build and integrate your own hardware sensors with ESPHome. There's a huge variety of video tutorials and "Top 10" lists on YouTube so I don't have to discover features purely by reading Home Assistant's frankly cluttered and somewhat poor documentation. Home Assistant is a bit rough around the edges sometimes and has a somewhat ugly UI but for the most part it fit the bill. Most importantly it was fully open-source and entirely self-hosted! It even has an open-source Android app that talks directly to my installation with no third-party intermediary! I already had a server running in my network closet downstairs so I went with Home Assistant Core installed in a Python venv and served by nginx. I slapped a subdomain on it, used Let's Encrypt for SSL and called it a day.

My current Home Assistant overview dashboard.

Finding IP-enabled AC switches turned out to be a much harder task than figuring out the software. I didn't really want to go with WiFi switches if possible due to the sheer amount of devices I would need to put on my network (44 games and counting!). WiFi also has weird latency issues depending on the distance something is from the nearest hotspot and can be unreliable at times. There really isn't much of an established Z-Wave ecosystem in the USA either. That seems to be popular mostly in Europe so most of the AC switches I found were for UK voltages and plugs. Zigbee switches in the USA seemed mostly centered around wall-mounted physical switches with remote control. Most of the products people used for Home Assistant plug-in control in America were cheap Chinese crap off of Amazon. I didn't want to risk a fire in my house, nor did I want to give Amazon a cent of my money if at all possible. There was an interesting product on adafruit that lets you build your own AC switches. Paired with an ESP8266 and ESPHome this looked like an excellent solution. However, the logistics of building 44 individual switches manually was daunting and I didn't see a good way to house the control electronics. There was also the Shelly 1 switch that looked promising. However, it was really meant to be installed in the outlet directly and it was also WiFi, so that was a no-go.

It turns out the solution was staring me directly in the face the whole time. My roommate had an AP-7900 rack PDU he got from his dad that he was using as a simple power strip. I've also used AP-7900s in embedded labs back at my first job and even coded some simple python control programs for them way back in the day. Their upsides met my needs perfectly: switched PDUs tend to use TCP/IP to query and control outlet state, they're enterprise grade and last for decades in datacenters and they're proven safe! The main downside, access to ethernet, was not a problem for me given I've previously run ethernet drops to every corner of the house. So I traded my roommate for a spare surge protected power strip I had in a closet and messed around with integrating the AP-7900 into Home Assistant. After finding a post on the Home Assistant forums where somebody else integrated one, I slapped together a template yaml file for myself and tested it out with my candy cabs in the basement.

A quick glance at eBay prices told me that outfitting all my games with AP-7900s was just not financially viable. They are rock solid and hold their value even decades after purchase. Also, I have my cabinets balanced on different circuits to ensure that no one breaker is ever loaded past 50% of its capacity. That meant that I had several clusters of games where there were only 3-4 total games to control. So, if I wanted to grab AP-7900s for the whole house, I would need around 20 of them. At about $125 a pop, that would run me at least $2500 just in the power switches. So I started digging into other PDUs that existed. I ended up finding out about the NP-02B 2-outlet PDU. I checked eBay and somebody was selling them for $40 a pop with fixed shipping. Bingo! Even assuming 20+ units, that's still around $800. While shockingly pricey compared to the cheap Amazon plugs it was still a massive savings and fit within my price range. So I bought one and tested out integrating it into Home Assistant.

After the initial success I decided to go all-in on NP-02B PDUs. I ordered as many as I could purchase from the guy on eBay selling them for cheap and got to work integrating them. Once I was happy with my settings I threw a repo up on GitHub with my yaml template files. I also wrote a simple python script that could fetch the outlet state or set the outlet state of any NP-02B regardless of firmware. This became necessary as only the oldest firmwares supported SNMP out of the box and its implementation was buggy. So, I needed to hit endpoints exposed by the HTTP interface to control the units. Even worse, the documented HTTP API only worked properly in half the units so I needed to reverse-engineer the other units. To make it easier to integrate with Home Assistant I encapsulated all of the quirks into the simple python scripts so that I could use the same template regardless of the PDU firmware.

A few of the NP-02B units that I ordered came non-working. The seller was super nice and sent me replacement units for free and said I could keep the old ones for parts. I opened them up and noticed they all had the same chip blown on the power supply board. Curious, I thought. But the back-and-forth with the seller revealed that he was also planning to list 30 more and I told him I would absolutely be buying again.

My initial stack of NP-02B PDUs before and after testing and provisioning them.

It was at this point that I had about 25 games integrated into my Home Assistant setup. This is when I started noticing problems. When I had only a few games integrated everything felt snappy like I expected it to. But, with 25 games, sometimes it took upwards of 10 seconds for a tap to result in the game turning on. In the meantime, you got no feedback that the system was attempting to power on the game and it felt really unresponsive. I started digging into the Home Assistant logs and at this point I realized that it was polling only one switch at a time! Furthermore, Home Assistant would only send state change requests when it wasn't actively polling. I checked and polling each outlet took about 250ms to 500ms depending on the outlet. Oof. Reading the Home Assistant documentation again, I verified that this was undocumented behavior. Going to the forums I found somebody complaining of similar issues and the response was basically "command-line switches suffer this problem, we won't be fixing this." Yikes! The command-line switch integration is advertised on their site as the preferred way of integrating with otherwise-unsupported devices! This wasn't something I could have known when I tested the first few games but it was definitely a dealbreaker. I'd already spent a few hundred on the project at this point, however, so it was time to get creative.

A major part of the polling slowdown was the network delay as well as the delay before the PDUs would send back their results. So, what if I created a daemon process that did the polling in the background and responded to Home Assistant instantly? In theory, it could work. So I went about taking the simple scripts that I'd made for the NP-02B and turning them into more of a full-fledged python project. I abstracted both the AP-7900 and NP-02B control and made a command-line interface that handled daemonizing as well as background caching. I swapped my yaml files over to use this utility and put up another github repo with the daemon program. Measuring the difference before and after, now Home Assistant could sweep the entire set of outlets in less than 2 seconds. I'd attained fast control again! Maybe not instant, but a worst-case delay of 2 seconds was way better than a worst-case delay of 10 seconds. Even better, my solution was available to anyone else that ran into this problem!

I ordered another chunk of NP-02B PDUs from eBay and got to work integrating them with the rest of my games. This went smoothly except for there were a few islands of games that didn't have access to an ethernet jack. For these, I figured the bandwidth requirements were low enough that I could throw some WiFi bridges under the games and make due that way. I grabbed a few bridge units that I had lying around from my previous house and wired it all up. The best one lasted 3 days before it disconnected from the WiFi. Even worse, when the outlets weren't reachable Home Assistant would pause for a long time waiting for outlets. The sluggish button problem was back in a bad way. I don't think its "done" to have to reach behind games and unplug/replug a WiFi bridge twice a week in order to turn games on. So, I made the tough decision to pull several more ethernet drops through the house. All in all I pulled five more drops inside the finished space and covered every remaining wall that didn't have ethernet on it. I also added two cables for the garage games that ran to a secondary switch in the furnace utility room. Some of the locations were not ethernet-drop friendly and required a great deal of creativity and drywall patching to finish.

A few select shots from the week of work it took to finish wiring drops.

With the connectivity problem out of the way I ordered more PDUs until I was able to hook up every game to my Home Assistant installation. All in all, I ordered 18 NP-02Bs and 2 AP-7900s. Including the third AP-7900 that I traded my roommate for I had 21 different PDUs controlling 44 games in Home Assistant. That turned out to be too much for Home Assistant's command-line integration to handle even with my daemon command-line utilities. So, it was back to the drawing board. From some research I figured out that the RESTful switch integration made use of parallel polling. I already had all the code in place to abstract controlling AP-7900s, NP-02Bs and even a NP-02 that got shipped to me by mistake. So, I added a second interface to it so that the core could be controlled via command-line or via REST and pushed the update to github. As an added benefit, I used the same core python library in my Naomi net dimm setup so that I could power-cycle games when sending a new one. Swapping all the config yamls for 44 games was tedious but well worth it! After the cutover, Home Assistant was able to poll all 44 games in less than a second and could respond to a power on or power off tap in a fraction of a second! Wow, way better than it could ever do even with the daemon that I wrote!

All my games available on the games dashboard in Home Assistant.

Remember the NP-02B units that came DOA? Where I thought that the broken units were just a fluke? Well, the problem ended up coming back in a bad way. A few weeks after I finished setting up all the games I realized that my Crazy Taxi wasn't responding to power-on taps. The power on button had also disappeared from my net boot web page, a sign that the PDU wasn't responding. I checked and sure enough it was completely dead. I popped it open on my bench and sure enough, the exact same chip had blown on the power supply. Curious, I started googling and that's when I found several reports of people talking about the same defect in their units. Well, shit. I'd just bought 18 of these things and apparently there's a known flaw in the power supply that causes them to kill themselves after several years. I checked and the manufacturer even acknowledged it, stating the current revision doesn't suffer from the flaw. Ugh, it was time to figure out how to fix them myself. I found what I thought was a compatible power supply board on Mouser, ordered a few, and then bodged one into the dead unit. Success! It was right back to life and after reassembly was controlling the Crazy Taxi like it never died.

The new power supply had the right pin configuration but was much smaller so necessitated a bodge. You can see the blown chip on the original power supply in the first picture.

At this point I had the fundamentals down. I had lightning fast control of my cabinets from any computer with an internet connection as well as through the Home Assistant app on my phone. The control was all local so I never needed to deal with AWS having a bad day or an ISP outage blocking me from turning on my games. All of the code was either open-sourced by me or an existing open-source project with a vibrant community. All of the switches themselves were safe, battle-tested and user serviceable if they failed. But I was missing one thing: guest control. And with that, I was missing the "wow factor." Having to be a turbo-nerd and go to some self-hosted website to turn on your games for your guests does not scream "cool" in the least.

I went back to the drawing board researching tablets and mounts. I figured surely there's a large ecosystem of people using Home Assistant, so there's gotta be a figured out solution to having a wall-mounted control surface. Turns out, nope! The people that did this cobbled their setups together with cheap tablets they found on Amazon or eBay and printed up atrociously ugly wall mounts. There was a lot of discussion about what tablets were even powerful enough to run the Home Assistant web frontend without lagging. There was the question of finding tablets that could be rooted so they could run in kiosk mode. The more I thought about it, the more I felt like tablets were a hack and not a real solution. I sat on the problem for about a month and considered designing my own hardware for the problem. And then I saw a tweet with a picture of an elgato Stream Deck.

This thing seemed perfect. It had actual buttons with haptic feedback. The screen seemed high resolution enough to have detailed photos and text on each button. It was self contained. It could probably be easily mounted on a wall. All it needed to drive it was a single USB C connection. Wow! I dug in and researched open-source alternatives to the official program and quickly found a python library that worked with all major models. I jumped on eBay, ordered one and got to work prototyping a Home Assistant front-end for it. What I ended up with is a python program that can talk to a Stream Deck over USB on any OS, takes as its configuration a list of entities and uses the Home Assistant local REST API to poll for state as well as toggle power for any configured game.

The initial prototype of my controlling software.

The only question that remained now was how do I install these things on the wall? They needed some sort of controlling computer and I didn't want to have to run a full desktop for each installation. The natural choice would be a Raspberry Pi. Problem is, they're pretty much a terrible fit for this sort of thing. They cost about 10x what they're worth right now, they're massively overpowered for what I need to do, they're bulky, power hungry and they have a tendency to corrupt microSD cards. So I started looking for alternatives and landed on the Rock Pi S. These things are basically perfect for small internet appliances! They don't even have video/audio out, but they do have built-in ethernet, WiFi and USB and they are dead simple to bootstrap. They're dirt cheap and readily available from the manufacturer's preferred distributor. They're also incredibly tiny and they run Debian out of the box!

These things are perfectly tiny!

Now, the only thing left was to drill holes in the Stream Deck base and mount it on the wall with toggle bolts. It turns out that if you mount the Stream Deck upside-down by rotating the mount, it sits at the perfect angle for operating a wall-mounted control panel. Even better, reusing thet magnetic base means that the Stream Deck snaps into place and can be easily removed for servicing. All of the Rock Pi S bits fit snugly inside the cable chase inside the base and I didn't have to design and print a holder! I installed the first Stream Deck upstairs to control the games that are on the main floor and used some D-line micro raceway to neatly tuck the ethernet and USB power cords away instead of leaving them dangling below the installation. Perfect!

The upstairs installation, with plenty of expansion room.

The second one I wanted to install on a wall in the laundry room that didn't have an outlet. Now, I'm no stranger to snaking power to new outlet locations so I was prepared to pop the ceiling tiles down and run romex all the way back to the breaker panel. But then I realized that I had some light switches on the same wall, in the same stud bay cavity, just on the other side of the wall! Perfect. A quick cut, an old work box, some fish tape and a little bit of romex later I had an outlet in the perfect spot to install the second Stream Deck. I imaged the microSD card from the first installation, changed the games in the config, set it up for WiFi and then installed everything onto the wall.

Outlet installation and the finished product.

After a six month journey I now have what I consider a "cool" installation. Guests can tap a button to turn on any game in the house, the installation looks slick, it's obvious how to use and I even get to monitor things on my phone. The outlets respond instantly to a power-on or power-off tap on my phone, computer or any of the Stream Decks mounted on the walls. Turning a game on using any interface instantly shows up on the other interfaces. Its locally-hosted and open-source across the board. The outlet hardware is documented and user serviceable, enterprise grade and battle tested by thousands of data centers. The candy cabinet power state seamlessly integrates with my net boot web interface. And even better, anyone else who wants can follow in my footsteps to set up a similarly slick setup! What more could I ask for?

Stream Deck Home Assistant Frontend Demonstration Video