Defining Your Home

Today we'll add another page to our dashboard.
We will dive deeper into JS and at first glance it will look more advanced and a bit messy - but it will be well worth the time, because we will implement a chat interface.
Let's build a bridge between our natural language processor (do) and our frontend dashboard.

I have done a lot of work on traditional LLM agents designed to execute custom scripts, and even writing it on it's own, mostly using Lang Chain.
I however, never gained any trust for it, and don't see much or any potential there to be honest.
My opinion on high power usage LLM agents for private smart home automations is that it can be achieved in cheaper, safer and faster solutions.

Why?

I wanted to build something that not only gives full access to all my (yo) scripts while on the run, but also gave appropriate responses, rendered beautifully right in the chat. Text to speech is of course crucial here as well. At first I was struggling to get a good voice for TTS - I needed it to sound just as good on desktop as on mobile, which seemed really hard to achieve since to my surprise I realized Apple has yet to release Siri on any open API. I ended up using what I already use on desktop, Piper and just create the wav server-side and playing the audio client-side.

My original thought for the chat was to build something that felt like ChatGPT - but faster, more reliable and without the thinking process. In simple terms:
Less thinking - more doing!

But before we start - I'll throw you a video teaser so you know what's what! (turn on your sound)

Ask What Time It Is

Understands 294 Million Phrases - This Is Not One Of Them

Shall we start?

First Thing First

Create a strong password and encrypt it within a file, and then set the path for the file.
This password will be used for the dashboard and the API later on in this article.

Since we will be executing shell commands from the chat, I added fun and proper authentication to the duckDash.nix module in form of a login screen - and protected the webserver with it.

If you are just following along this article and using my modules, you don't have to do this login page - as it is already implemented into the duckDash module, I just wanted to show it off - since security is major key.
And damnit, I won't lie.. I just love these falling ducks. They really make me smile every time I am prompted for the password.

This Might Require An API?

This might not be a requirement, but I felt it was the way to go for me - if I was not going to need it now, I most likely would need it later in some form.
So I wrote a Rust service with a endpoint for the `yo do` (the NLP) command, some zigbee device control endpoints and filled it with some other endpoints that I might need in the future, and password protected them, so no sketchy intruders can use them.

Now when the API service is up and running, and we have proper authentication on both the dashboard login and the API, we can finally continue creating our next page (tab) on our dashboard - the chat interface!

As Previously Shown...

Adding pages to our dashboard follows a simple structure:
Page number, title, icon for the tab, give it some styling with CSS, symlink required files and insert page code.
And Voilà, you got yourself a page!
Here's how I did:

I know that must have been a lot to take in, and I do appologize for the code structure, as I can barely see code organization is my biggest enemy.
If you had trouble following along in the code there, I suggest turning on the duck commentary and let the ducks guide you through it.

Conclusion

So what did we end up with?
Good question!

We now have a chat that you can communicate with using natural language, depending on your yo scripts, you can ask it questions, tell it to execute commands like control your lights, TV's, ask about your calendar events, check when your local train departures, check how your local sports team is doing etc etc.
As I am writing this I have 60 scripts available for use within the chat, so the possibilities are pretty much endless.

I can't see myself using it really, but I do have a command to trigger local Ollama response too.
The chat response bubbles automatically renderes any possible tables or structured layouts as markdown for best possible look.
It can even play media, like music or movies right in the chat response bubble. The chat will automatically recognize .m3u/m3u8 or video URL's from the yo scripts output and play them in the chat bubble.

When a chat response is recieved it will automatically play the /tts/tts.wav file if it has been modified within the last 30 seconds.

I also included a attachment button so I can send photo's or documents from my phone to store for backup on my server.

Clicking a chat bubble will pop it into full-screen mode.
Here is one more demo video asking for a music playlist to be played on the TV.





So what do you say?
Goodbye, traditional LLM chatbots?

The Full Source

Keep Reading