In search of the perfect experiment runner, and why I will most likely have to write my own

For a long time, I have been mildly dissatisfied with how most data acquisition code in research ends up being written.

"Mildly dissatisfied" is, of course, not quite true. The real issue is that acquisition code tends to start its life as a quick script and then, through neglect and necessity, quietly graduates into infrastructure. By the time one realizes this, it is already too late and the script has grown to have instrument drivers bolted onto it, configuration values scattered throughout, and just enough (implicit) assumptions to make rerunning the measurement a month later slightly painful and six months later nearly impossible.

I have run into this often enough that I keep returning to the same thought - there ought to be a good boilerplate for experiment execution. Not a giant framework. Not a workflow engine. Just a clean and disciplined way to run measurements that makes the sensible thing the easy thing.

I have looked for this more than once, and each time I end up in roughly the same place - close enough to be annoyed, but not close enough to actually use.

The actual problem

The problem is not collecting numbers from instruments. That part is usually straightforward enough. Import a (self-written, more often than not) VISA-driver, figure out the address of the device, and pray that the driver has implemented the functions you need.

The problem is everything around the numbers.

Who ran the experiment? On which machine? With which code revision? Which parameters were swept? Which instruments were connected? What happened before the run failed halfway through? Which functions were called to derive the values that eventually made it into the saved dataset? How long did the calibration step take compared to the acquisition itself? Which version of the script produced the plot that ended up in the Teams chat or in the draft of a paper?

In a well-run experiment, these questions should not require memory. They should be answered by the output of the experiment itself.

Too often they are not.

Instead, one finds a CSV file with a timestamped filename, perhaps a PNG, perhaps a notebook, perhaps a half-helpful comment somewhere saying "this one looked good". This is not reproducible science.

Why I care about this

Part of it is simply that I do not enjoy doing the same work twice because the first version was not logged properly.

But more than that, bad acquisition code accumulates technical debt[1] unusually quickly. It tends to sit exactly in the land between hardware, software, and metadata. That makes it fragile in a very particular way. A measurement can appear to work perfectly well while quietly failing to record the only contextual information that will matter later.

This matters in research, obviously, but not only there. The same script often ends up being reused by a student, adapted for a different setup, or dusted off after several months when nobody remembers quite how it was supposed to be invoked. At that point, the difference between "a Python script that runs" and "an experiment runner becomes very real.

A runner should not merely execute code. It should leave behind a coherent trace of what actually happened. It should re-tell the experiment's story.

The requirements

My requirements are not especially exotic. That is partly why this is so irritating. It should be possible to do this elegantly, but for some reason^[2], I still rely on ad-hoc scripts.

Python

Python is the obvious choice here. Not because it is perfect, but because it is the path of least resistance in most labs. People already know enough of it to be productive, instrument libraries already exist, and it remains the default glue language for this sort of work.

If a system is meant to be reused by students, colleagues, or my future self in a hurry, Python is hard to argue against. Plus, all dependencies I would need are already written in Python.

Rich metadata must be logged

This is not optional.

At a minimum, I want operator name, machine name, timestamp, experiment name, version, a free-text description, and preferably information about the execution environment (CPU/mem/IP) as well. This is not just me being a tech geek wanting to know all there is about the system because rich logs are cool - no, as the one who generally manages computer infrastructure in the lab, knowing the machine configuration can sometimes give me more information to piece together a timeline of when an experiment was run than a single anonymous timestamp does. At least from a configuration mangement perspective it does. In many cases I would also want code revision, instrument identifiers, and perhaps some snapshot of relevant configuration/calibration files, or at least file hashes.

The point is not to log metadata because it looks professional. The point is that metadata is often the only thing separating a meaningful dataset from an "orphaned" one. I'm forgetful (or involved in too many experiments maybe?), and the only way I can with certainty recall what the dataset means is by having all these contextual clues to backtrace. Plus, more than once I have been burned by forgetting to log something which I thought I'd remember (an assumption, a reason for changing parameter X from x0 to x1, etc).

Program execution should be logged implicitly

This is the part I find many systems get wrong.

If logging relies entirely on the discipline of the experiment author, it will eventually fail. Not because the author is lazy, but because they are busy thinking about the experiment instead of narrating it to a logger. Once or twice I have been meticulous in logging every little detail of the execution, but I do miss things, and besides, explicit log calls (logger.info(f"Initializing waveplate {id} to {theta} degrees")) clutters up the experiment code, making readability low. My most recent experiment runner contains ~130 explict logging calls in a single file, to give an example.

The code should therefore produce a trace almost by default of function calls, arguments where reasonable, important checkpoints, exceptions, warnings, start and stop events. Not in a way that turns the output into unreadable noise, but enough that one can reconstruct what happened without guesswork.

If a particular step significantly affected the measured data, I want that to be visible.

Timing between user-defined checkpoints should be easy

This is one of those things that sounds minor until one has needed it enough times.

Often the most useful performance information is not total runtime, but the time spent between meaningful stages like instrument initialization, warm-up, calibration/optimization, sweep execution, post-processing, export. This is useful both for optimizing the code, to plan long-running experiments (typically overnight runs) and for sanity checking.

If a gradient-descent polarization aligner suddenly takes ten times longer than usual, that is often worth knowing even if the run technically completed. It gives valuable information about convergence, and whether a particular experiment is executing "reasonably sanely".

It should be boilerplate-friendly

This is perhaps the most important point.

I do not want a framework that requires its own philosophy. I want something that can be cloned, understood quickly, and adapted without immediately having to plunge into the docs. The barrier to starting a new experiment should be low, but the quality of the resulting structure should be high.

Ideally, the shortest path should also be the disciplined one. It should be easy to do the correct thing.

What usually goes wrong

In my experience, acquisition code usually fails in one of a few predictable ways.

The data format is too thin

The raw numbers are stored, but little else. That feels fine until one actually wants to compare runs, reproduce a measurement, or hand the dataset to someone else.

Questions I often return to are akin to

• "OK, the interference curves look good, but what was the integration time?"
• "The estimated mean photon number is plausible, but what was the peak count rate - did the detectors saturate?"

The control flow is too ad hoc

There is a for loop, maybe a few try blocks, perhaps some instrument setup at the top, and an export at the end that may or may not run if something fails halfway through. This is tolerable in a 30-line script and increasingly unpleasant beyond that.

The experiment logic and the bookkeeping are mixed together

This is perhaps the biggest one. The actual scientific or engineering logic gets buried in code for file naming, logging, export, plotting, cleanup, notifications, and exception handling. The scaffolding needed just to enable decent logging, plotting, and data export easily takes up a few hundred lines of code already. The result is that the script becomes harder to read precisely because it is trying to do the right things.

A decent runner should separate these concerns without making them cumbersome.

How I would probably implement it

My current instinct is that the right abstraction is something context-based.

The experiment run itself should live in a context manager that handles setup and teardown properly, records start and end state, catches failures cleanly, and ensures that partial results are not silently lost. This is not particularly glamorous, but it is exactly the sort of thing context managers are good at.

Context-based semantics

An experiment should have a clear lifecycle: enter, run, store, finalize, exit.

That makes graceful exits much easier to reason about. If the user interrupts the run, or an instrument throws an exception, the framework should still have a well-defined opportunity to flush logs, save partial datasets, serialize metadata, and execute any post-run hooks that are safe to run.

A failed experiment is still an experiment. It should produce a useful record.

Structured metadata and parameters

Metadata and parameters should be explicit objects rather than informal dictionaries (in the best of cases, more often than not - simply global variables) sprinkled throughout the script.

That gives them a proper home and makes validation possible. It also nudges the user toward writing down what the run actually is before they start talking to instruments.

Tracing as an annotation, not as a habit

If function tracing is useful, it should be easy to opt into with something like a decorator (@Traced). That strikes me as preferable to manually writing log statements everywhere. It keeps the experimental code readable while still making important transformations visible in the trace.

One probably has to be somewhat disciplined here, because logging every function call in a scientific Python stack is an excellent way to create noise. I see this in my own monolithic logs that I run at the finest level of detail. I get a complete picture of the execution, but at the cost of losing parsing speed. Perhaps disciplined use of log levels can mitigate this, together with a capable log viewer? But for user-defined experiment logic, it can nevertheless be very valuable.

Storage should happen incrementally

This is another thing I have learned to distrust: any design that waits until the very end to write the important output. Far too often have I made the mistake of storing measured values in a list during the experiment, and only writing to file after a complete run. This has two important consequences:

• Intermediate / incomplete results cannot be monitored during the experiment. That is, there is no way to "peek" at how the data acquisition is going. Bad for long-running tasks that may or may not fail (interferometer drifting out of alignment during a multi-day run), requiring manual intervention.
• If the program crashes, all data is lost. Matter of fact is that computers reboot unexpectedly, a VISA timeout may occur, or a simple parsing error that only appears mid-run can ruin many day's progress
• For large runs with large amounts of data measured, you run the risk of running out of memory.

Results should be persisted as the experiment runs, or at least buffered in a way that makes partial recovery possible. A run that fails at 98% completion should not leave behind nothing except a stack trace and some regret.

Post-run hooks are useful

Sending an email, zipping the result directory, exporting a summary, generating a quick-look plot, or copying artifacts somewhere sensible. These are exactly the kinds of things that are annoying to reimplement in every project, yet useful often enough to deserve a place in the structure.

The key is that they should feel like hooks, not obligations.

What the ideal workflow looks like

In the ideal case, the experiment code should read like the experiment itself.

That means instrument setup should be obvious, metadata should be declared up front, parameter sweeps should be explicit, and the body of the run should mostly consist of the actual sequence of operations. The runner should take care of the scaffolding around it.

Something along these lines would be close to what I want:

import numpy as np

from qexp import Experiment, Parameters, Metadata, EmailPostHook
from qexp.annotations import Traced
from qexp.instruments import TektronixFunctionGenerator, ThorlabsPowerMeter

fgen = TektronixFunctionGenerator(<VISA_ID>)
pm = ThorlabsPowerMeter(<VISA_ID>)

meta = Metadata(
    author="Joakim",
    experiment_name="Hong-Ou-Mandel",
    experiment_version="0.1",
    experiment_description="A first HOM experiment using the new detectors",
)

params = Parameters()
params.add("VOLTAGES", np.linspace(0, 10, 100))

hook = EmailPostHook([<recipient1>, <recipient2>])

@Traced
def calc_something(val1, val2):
    # Function call is implicitly logged along with passed args
    return val1 - val2

with Experiment(meta, params, zip_results=True, post_hook=hook) as exp:
    for v in params.get("VOLTAGES"):
        fgen.set_voltage(v)
        p = pm.get_power()
        exp.store_reading("POWER", p)

        calc_something(v, p)
        # ...
        # Generate intermediary plots?

# Upon exit, data is zipped and the post-hook is executed

Paving the way to scheduled job running

When deploying the modest, wildly outdated, but beloved workhorses of the lab - our HP ProLiant G7's, I was reading up on how HPC manages job scheduling. There I came across SLURM, and the philosophy of job submission and resource management. That got me thinking "hey, I should write something similar for experiments in the lab". I thought that it would be neat to have one server that has control over a set of VISA instruments, and to which users submit their experiments to be run. As the server has control over running jobs, any jobs that can run in parallel (i.e. non-clashing resource requirements) would be ordered to do so, with automatic retries, handling of priorities etc.

A unified, or at least standardized, experiment semantic like the one above could probably be used for this too.

So, what's next?

Thing is, the effort to implement the wishlist above would probably take a non-negligble amount of time. Pragmatically, the hundred-lines-of-log-call code is probably the most time efficient to rely on for the moment. Science never sleeps, and results are all that matters, so unless there is a considerable gain in implementing this, this will have to be put on the back burner. Perhaps it could be formulated into a project for our excellent bachelor's students!

[1]: A term that was ingrained in me during my undergraduate software engineering classes, but whose true meaning I have only come to fully grasp once it hit me like a bullet train. [2]: Foreshadowing for another post?