How to store arrays¶

A box can hold arrays too, such as an image from a camera or a block of samples. An array field has a fixed shape and element type, and it takes arrays from numpy, torch or any other library that can hand its arrays over through DLPack or the buffer protocol, as long as they are in CPU memory. This guide shows how to declare one, how to read and write it, and the two ways to save a copy when the arrays are large.

1. Declare the field with a shape and a dtype¶

Put a Shape and a DType in the annotation. The type you annotate decides what a read gives you back: a numpy.ndarray, a torch.Tensor, or, with SupportsDLPack, an array object of sharedbox's own that any DLPack library can take:

from typing import Annotated

import numpy as np

from sharedbox import DType, Shape, SharedBox, SupportsDLPack


class Sensor(SharedBox, name="example-sensor"):
    frame: Annotated[np.ndarray, Shape(4, 6), DType("uint8")] = np.zeros(
        (4, 6), np.uint8
    )
    raw: Annotated[SupportsDLPack, Shape(3), DType("float32")] = np.zeros(3, np.float32)

2. Write and read an array¶

You assign and read an array field like any other:

sensor = Sensor()
sensor.frame = np.full((4, 6), 7, np.uint8)
print(int(sensor.frame.sum()))  # 168

A write copies the array into the box, and a read copies it out into a new array, so changing your array afterwards never changes the box, and the other way round. For a large array, those copies and the new memory are what you pay for; switch between the three ways to see where each one goes:

Where the bytes go

A write copies your array into the field. A plain read copies the field into a new array, which needs fresh memory every time.

read_into copies the field into an array you already have, so no new memory is needed.

writing hands you the field itself. Your code changes the bytes where they live, so nothing is copied.

3. Read into an array you already have¶

A plain read gets fresh memory for a new array every time. If you read the same field again and again, for example once per frame, pass an array you already have to read_into instead. It copies the field into that array and returns it:

frame = np.empty((4, 6), np.uint8)
sensor.read_into("frame", frame)
print(int(frame.sum()))  # 168

The array has to be writable, laid out in one block in C order (C-contiguous), in CPU memory, and of the field's dtype and shape. On Windows, getting fresh memory for a large array costs far more than the copy itself, so reading into an array you keep is several times faster there.

4. Fill an array field in place¶

If your data comes from something that can write straight into memory you hand it, such as a camera driver, writing saves you the extra copy. It gives you the field itself to fill:

with sensor.writing("frame") as frame:
    frame[0, :] = 255
print(int(sensor.frame[0].sum()))  # 1530

The array starts out holding the field's current value. When the block ends, the box counts the write and wakes everyone watching the field. That happens even if the block stops halfway with an exception, and then the field keeps whatever the block wrote before it stopped.

Everyone else waits while the block runs

The block holds the box's write lock, so every read and write of the box waits until it ends, in every process. A reader that waits longer than its lock timeout, 5 s unless the class sets lock_timeout, raises LockTimeoutError. Do the slow work before the block, and keep the block itself short.

The array is yours only inside the block

Writing to the array after the block takes no lock and tells nobody, so a reader can see half of your change. Use it inside the with block and nowhere else.

5. Read with another library¶

A SupportsDLPack field reads back an object any DLPack library imports:

raw = sensor.raw
print(np.from_dlpack(raw))  # [0. 0. 0.]

To read fields back as an array type other than numpy's and torch's, call register_array_type with the type and its from_dlpack function before you define the box class.

6. Handle the wrong shape or dtype¶

try:
    sensor.frame = np.zeros((6, 4), np.uint8)
except ValueError as error:
    # Sensor.frame holds an array of the field's Shape; the value's shape differs
    print(error)

If you assign an array of the wrong shape, you get a ValueError; the wrong dtype, or a value that isn't an array at all, raises TypeError. Either way the field keeps its old value. For what a large array costs to read and write, see How fast a box is.

The whole script

"""The script of the guide "How to store arrays"."""

from typing import Annotated

import numpy as np

from sharedbox import DType, Shape, SharedBox, SupportsDLPack


class Sensor(SharedBox, name="example-sensor"):
    frame: Annotated[np.ndarray, Shape(4, 6), DType("uint8")] = np.zeros(
        (4, 6), np.uint8
    )
    raw: Annotated[SupportsDLPack, Shape(3), DType("float32")] = np.zeros(3, np.float32)



sensor = Sensor()
sensor.frame = np.full((4, 6), 7, np.uint8)
print(int(sensor.frame.sum()))  # 168

frame = np.empty((4, 6), np.uint8)
sensor.read_into("frame", frame)
print(int(frame.sum()))  # 168

with sensor.writing("frame") as frame:
    frame[0, :] = 255
print(int(sensor.frame[0].sum()))  # 1530

raw = sensor.raw
print(np.from_dlpack(raw))  # [0. 0. 0.]

try:
    sensor.frame = np.zeros((6, 4), np.uint8)
except ValueError as error:
    # Sensor.frame holds an array of the field's Shape; the value's shape differs
    print(error)

sensor.close()
Sensor.unlink()