Command output:
chunk_start_samples,
^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
to_cpu=_peeler_process_context.ctx.to_cpu,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/opt/venv/lib/python3.13/site-packages/dartsort/peel/peel_base.py", line 540, in process_chunk
peel_result = self.peel_chunk(
chunk,
...<5 lines>...
**peel_kw,
)
File "/opt/venv/lib/python3.13/site-packages/dartsort/peel/matching.py", line 268, in peel_chunk
chunk_template_data = self.matching_templates.data_at_time(
t_s=chunk_center_seconds,
...<5 lines>...
resid_offset=resid_offset,
)
File "/opt/venv/lib/python3.13/site-packages/dartsort/peel/matching_util/drifty.py", line 350, in data_at_time
self.spatial_at_time(t_s=t_s)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "/opt/venv/lib/python3.13/site-packages/dartsort/peel/matching_util/drifty.py", line 329, in spatial_at_time
pconv = full_shared_pconv(self.b.tconv, normsq_spatial_sing)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
File "/opt/venv/lib/python3.13/site-packages/dartsort/util/spiketorch.py", line 272, in full_shared_pconv
n_units, rank__, chans = spatial_sing.shape
assert rank == rank_ == rank__
out = spatial_sing.new_empty((n_units, n_units, up, conv_len))
~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
spatial_sing_flat = spatial_sing.view(n_units * rank, chans)
tconv_flat = tconv.view(rank * rank, up * conv_len)
RuntimeError: CUDA out of memory. Tried to allocate 32.55 GiB. GPU 0 has a total capacity of 14.56 GiB of which 13.47 GiB is free. Including non-PyTorch memory, this process has 1.09 GiB memory in use. Of the allocated memory 137.93 MiB is allocated by PyTorch, and 804.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
Hit an error. Writing traceback to /tmp/nxf.FARSJehkXL/capsule/code/../results/dartsort_experiment1_Record Node 101#Neuropix-PXI-100.ProbeA-AP_recording1/experiment1_Record Node 101#Neuropix-PXI-100.ProbeA-AP_recording1/traceback.txt. work_in_tmpdir was true, so the files won't be kept. Set save_everything_on_error to keep them.
Another failure (after #57). Is the peeler supposed to use that much GPU RAM?