Skynet: LLMs controlling real robots and drones with Bash

I have been playing with using machine learning to control physical devices for a while now. And lately there have been so many interesting things happening!

My friend and collaborator Philippe Charrière started working on a lightweight Bash library for interacting with LLMs called Osprey. When I found out about it, naturally I took the nerd bait.

The result is Skynet, a lightweight command line interface for LLMs to control real robots and drones using Bash.

What can it do? Check out this video showing Skynet controlling a Parrot Minidrone using Bluetooth:

Skynet can run entirely on your local machine using Docker Model Runner or Ollama. It can also be configured to run using remote services.

All about the context

What makes this now possible is the Model Context Protocol (MCP) a new standardized way for language models to obtain information from external systems and perform actions.

You probably have already been hearing about MCP everywhere. Just in case you want to know more, I happened to write a post "Calling MCP Servers The Hard Way" myself a few months ago.

An MCP server can expose tools, which allow models to discover and perform actions by calling these tools. Each tool defines an operation it can perform, along with a description, and any inputs and outputs. A model can then request that a tool be called if it needs it, based on the context used.

For example:

💬 take off                                                               
                                                                                                                                                     
🛠️ calling tool 'takeoff' on http://localhost:9090 with {}                                                                                           
✅ result {"tool_name": "takeoff", "content": "minidrone taking off"}

So when you tell the drone to "take off", since the model is aware of the "takeoff" tool, it can request that it be used to fulfill the user's request.

Skynet archtecture

Skynet itself is a Bash script. Under the hood it uses standard command line tools like curl, jq, and awk to call the LLM model server, and also to make any tool calls to available MCP servers for hardware devices.

Why Bash? Bash is the "native language" of the Internet, already installed on millions of systems across the planet. What better choice for a robotic takeover?

MCP and robots and drones, oh my!

There are other devices that have MCP Servers available. Here is another video, showing a Wowwee MiP robot:

The MiP and the Minidrone both happen to be devices that are controlled remotely via Bluetooth. They also happen to be devices that have support via the TinyGo Bluetooth package. That made it relatively straightforward to write MCP servers for each of them, thanks to another Go package called mcp-go.

Here is the Go code for mcp-mip and the Go code for mcp-minidrone, respectively.

There are other MCP servers to physical devices in the wild, written using the myriad programming languages that have MCP support, with more appearing each week.

What happens next? Find out at Gophercon in New York

So what happens next? Perfect timing to ask that, since we are getting ready for the latest edition of the GopherconHardware Hack session coming up in NYC in exactly three weeks. Come and play with us there, or follow along from your own hidden rebel base.

I will write more about Gophercon in an upcoming post, as we unleash the machines in the Big Apple.