Years at a supercomputing center taught me one thing: No one knows...

84 views Jul 15, 2026 ~3 min read

Years at a supercomputing center taught me one thing:

No one knows what the f*** they're doing.

So I've decided to simplify the process so you can say "i'm out" more quickly!

A quick introduction to the horus-runtime github.com/temple-compute…

                            2
                        
Some background:

I'm a computational chemist / software engineer and I've been grinding lots of hours configuring environments, learning about SLURM and waiting hours for a compute queue that never gets free.

And one thing that I hate the most: wasting time.

                            3
                        
Why can't we just ask the HPC "do this"? When doing research, it should be enough to know the field your'e working on. No need to spend hours learning commands that probably only work on a compute node with the oldest version of miniconda installed.

                            4
                        
After playing a little bit with some workflows managers like Galaxy or Nextflow, I realized I had fell into the "another tool, another day lost reading a manual" trap. And it was not enough just using the  tool, I had to still manually install environments or pull images.

Then I saw it clearly, why workflow engines are just a collection of chained commands that I can just run into a bash script? I need them to do more work for me.

And thats basically how Horus was born: a heaven-sent workflow engine that automatized the whole process.

                            6
                        
Originally though for life-sciences related workflow, now is expanding to all compute-intensive tasks:
- AI models training / inference
- Physics-based simulations
- Drug discovery / protein engineering
- Climate modelling
- Cryptomining 💸

But how could I escape the "just another workflow engine" trap? Let's dive a little more on why the horus-runtime is radically different to any other workflow engine you may have seen before.

Heres a simple diagram on how the engine is structured:

One of the most important things I will highlight is that all of these components work per-task, not per-workflow. Thus making the engine extremely flexible.

For example, a quick executor swap may execute the exact same runtime in two different "environments".

This sounds trivial but it is super helpful when building reproducible workflows: the execution layer gets embedded directly within the workflow.

Now imagine that I have an underused machine with a GPU, and that I want specifically this task to be run there:

Because both the local and SSH executors work using the target protocol methods, the swap works "for free".

But... what happens with inputs and outputs? The task will need the input files in the target machine (common sense).

Let me introduce: transfer strategies.

                            11
                        
Horus automatically picks the fastest way to move artifacts between targets.
- Local > no-op.
- Same cluster > direct copy.
- Remote > whatever transfer strategy is available.

                            12
                        
Cool... but my lab uses some absurd niche tool nobody has heard of.

Can Horus run it?

Yes.

Horus has a plugin system. Everything you've seen so far (artifacts, runtimes, executors, targets) are just the builtin plugin implementing the core interfaces.

                            13
                        
Plugins aren't just for applications.
In Horus, almost every part of the runtime is pluggable.

Examples:
• Python environments
• SSH targets
• SLURM executors
• Logging backends
• Event transports

If you've made it this far, I'd encourage you to take a look at the Horus SDK documentation:

docs.templecompute.com/sdk

We're also building a community workflow repository so you don't have to start from scratch:

github.com/temple-compute…

Contributions are always welcome.

And of course, we're building a UI around all of this.
It's heavily inspired by the design principles we've been developing: making complex workflows feel simple.

If you're curious about the UI or just want to know more, check it out at templecompute.com

                            16
                        
If any of this sounds interesting, I'd love your feedback.

Try it, break it, open an issue, submit a PR, or just tell me why you think it's a terrible idea.

Happy workflowing!

Years at a supercomputing center taught me one thing: No one knows...

Actions

What You Can Do

Actions

What You Can Do

More in Tech & Development