UI Idea: control the jumperless from a webcam feed

The wokwi bridge is great and all, and typing bridge connections manually is certainly fun as well.

But I believe better things are possible!

So, I’m thinking about what a jumperless specific UI could look like.

This is one idea I’m playing with:

Setup

Point a webcam at the jumperless (ignore the stepper motor, it’s just a weight)

UI

The UI shows the webcam feed. To interact with the jumperless, simply click on things and connect them.

Since this may be hard to imagine, I’ve made a small PoC, which shows:

  • how to map elements on screen to the picture (by identifying the 4 corners of the board)
  • that elements are interactive (currently only selecting them is possible)

Sources: GitHub - nilclass/jumperlab: TBD
Deployed version: React App

Next steps:

  • actually communicate with the Jumperless (maybe via WebUSB?)
  • retrieve netlist and show connections on screen
  • controls to add / remove bridges

Future ideas:

  • Add probes, to show voltage (and current) live on screen
  • Detect the position of the board automatically: I don’t know enough about feature recognition to judge how hard this would be. If the UI has controls over the LEDs, one way to do it might be to selectively turn on some LEDs and detect their position?
1 Like

Holy shit, that is so fucking sick!

And that POC is great, I can actually see that being a really nice was to use a Jumperless. And it really doesn’t look like it will be that difficult to get this to a place where it’s useable.

Let me know what parts you’d like help with. Now that you’ve made a choice for the framework, I’ll try to get comfortable with React.

I could even just make a version of the bridge app that pulls from this instead of Wokwi. That’s the quick/lazy way until we figure out WebUSB.

Thanks for this!

Thanks for your feedback, glad you like it too :smile:

I could even just make a version of the bridge app that pulls from this instead of Wokwi. That’s the quick/lazy way until we figure out WebUSB.

That would be difficult right now, since there is no server-side to pull anything from.

Here’s a sketch of the architecture I had in mind:

(instead of WebUSB, the Web Serial API seems like a better choice – still, both are experimental and not supported by all major browsers, which is a bummer)

For comparison, the architecture with Wokwi looks more like this:

(at least that’s how I understood it, please correct me if this is not correct)

To avoid having to deal with the Web Serial API for now, I’m taking a different approach though: I’ve written a small CLI tool, “jlctl” that communicates with the jumperless via the serial port. It can parse the netlist output (from n), and produce node files (applied with f).
That tool also acts as an HTTP server, which the Jumperlab UI can communicate with.

So, the topology for that scenario would be more like this:

Now that you’ve made a choice for the framework, I’ll try to get comfortable with React.

Enjoy, React can be fun! Don’t take too much inspiration from the code I’ve written though – it’s a somewhat crappy prototype at this point :stuck_out_tongue:

Let me know what parts you’d like help with.

In general, a stable machine-readable serial protocol would be great. It could even be mixed into the current interface, if the machine-readable parts are easily identifiable (e.g. lines prefixed with ::, followed by structured values). That way the modification to the firmware would be minimal. Any host-side code can just throw away all lines not starting with :: and interpret the remaining ones.

Another thing on my mind is access to measurements, like the ADCs on the RP2040 and the current sensors. One way this could work from an interface perspective is that the host-side “subscribes” to a measurement: from then on the jumperless periodically sends those measurements, until the host-side “unsubscribes”. The period can be hard-coded for now, and configurable later.

These are just random ideas though. Let me know what you think

Dude, that CLI tool is amazing. What you’re doing is exactly what I was hoping would happen, someone who can program real computers getting nerdsniped into starting a desktop app in a modern language.

You’re spot-on with the current architecture, and the one you proposed is what I had in mind for the next step as well. Having control of the whole stack would let us do a lot more cool shit.

That’s a great idea, and it’s pretty easy to implement.

What I’m going to do first, is to document the existing protocol as it currently is.

Then I’ll write another document for the proposed interface for machine-readable interface. And there’s no reason to only do connections, we can give it control of any step in the chain of:

  1. Parsing input to bridges
  2. Placing bridges into nets
  3. Finding paths through crosspoints
  4. Choosing LED colors for nets
  5. Sending Data to the crosspoints

Because I see you store stuff as netlists in your CLI, we might as well just take those in as a netlist instead of converting them to bridges and back and hoping the Jumperless makes the same choices. What’s cool is then you have control of the color of each net.
And it should run faster on a real computer.

For the measurements, my first thought would just be a call-response setup, maybe with some fields for: how many measurements to return (or until I say stop), time between measurements, samples per measurement, (maybe do a “what to connect to for this measurement” but that might just be better to leave to the netlist).

I’ll look into those. Right now I’m using pySerial and I spent a lot of time manually managing the connection to make it pretty robust, like you can flash firmware to the Jumperless with the bridge app running and it’ll just reconnect when it’s done like nothing happened. I wonder if there’s a way to just port that logic over.

I’m still kinda sketched out by those WebUSB things, every time I install Arduino’s online editor web client (I was considering it for Jumperless), it would break something in my computer’s USB controller and the USB ports wouldn’t work until I reinstalled macOS.

But that may be different now. idk.

Anyway, here’s the off the top of my head new protocol:

::bridgelist[1-2,5V-3,gnd-25]
::netlist[{"index": 1,"name": "GND","number": 1,"nodes": "GND,17","bridges": "{GND-17}","color":"0x493200"},  {"index": 2,"name": "+5V","number": 2,"nodes": "5V","bridges": "{0-0}","color":"0x0023aa"}] //I just copied yours, I might make some fields optional and allow it to be written differently to follow the style of the others
::lightnode[25,2,4,33 : 0x503000]
::lightnet[8 : 0x503010] //maybe accept names too
::getmeasurement[adc0 : "measurements" = 50, "interval" = 10, "samples" = 50] //maybe allow multiple measurements and they'll be returned interlaced with a header saying what it's from
::gpio[0 : {"high", "delay" = 10, "low", "delay" = 10} : "loop" = 100 }
::uart[send : "I'm sending data\n"]
::uart[receive : 100 : "timeout" = 500] //or 0 for until there's no more data
::arduino[upload : "//compiled hex firmware"]

Let me know if you think anything else should be included in that list or any changes you think would make your life easier.

I’m so stoked your working on this, because now I have something to work on too.

1 Like

What you’re doing is exactly what I was hoping would happen, someone who can program real computers getting nerdsniped into starting a desktop app in a modern language.

You got me there :stuck_out_tongue:

Regarding robust serial connection: very good point, I’ve made some changes to jlctl to deal with it there as well.
Now jlctl tries to determine the port automatically. In server mode, if the port gives an error, it re-initializes it. This works well when reconnecting the board, even if the path of the port changes in that process (which it does - at least on Linux - whenever the old port is still “in use” because something has a connection open).
Let me know if that works on Mac as well, I haven’t had a chance to test it elsewhere (jlctl list-ports should list all USB serial ports, and identify two of them as “JumperlessPrimary” and “JumperlessArduino”).

For the measurements, my first thought would just be a call-response setup

Totally fine too - that actually makes it easier to integrate for me as well at this point.

btw, somewhat related to that: the RP2040 supports running it’s ADCs in free-running mode, where one or more of the ADC inputs are sampled repeatedly in a (configurable) interval and written to a FIFO. That FIFO can be used as a DMA read target, so it’s possible to use DMA transfers to shovel those readings into a buffer without CPU intervention. Only once the buffer is full the CPU needs to get involved again and do something with it, e.g. copy it over to a USB bulk endpoint’s buffer, and start the next transfer. Together with some trigger logic (maybe running on the second core?) that would almost be an oscilloscope already.
Anyway, … it’s more of a future idea :smile:

Anyway, here’s the off the top of my head new protocol:

There’s some great ideas in there!

To keep things easy to parse, we could stick with just positional attributes for now (no key/value pairs).

Like:

// set color of net number 8
::lightnet(8, 0x503010)

The netlist could also be split into multiple messages if that’s easier

::begin_netlist
::net(1, "GND", 1, "GND-17", "{GND-17}", 0x493200)
::net(2, ...)
::end_netlist

uart[send] / uart[receive]: I like the idea! Would this be an alternative to having the second UART port, or would both be available and it’s up to the user to only use one at a time?

Extracting the bridges from the netlist is not a big deal, so bridgelist isn’t a high priority for me.

Now that the connection via jlctl is working, I’m refactoring the frontend PoC into something more extensible and less brittle, then I’ll add support for making basic connections in there, to have a first prototype that is actually doing something.

Once I’m there we can think about what to add as a next feature - that also depends on what’s possible with the protocol at that point.
“Single ADC measurement” and “setting node (or net) color” both sound like good candidates.

1 Like

Update

Jumperlab now has:

  • a README, with basic instructions for how to run it
  • a hastily drawn logo, mainly to get rid of the react default favicon
1 Like

Siiick. I really like the logo.

Note to users: do not pour bubbling purple chemicals onto your Jumperless. I haven’t tested how it responds to the other colors of bubbling chemicals yet, but I know it doesn’t respond well to purple ones.