🛠️ Train 300M/32-Layer Model in 1.5GB RAM on Base M1 Mac
Summary
Ullis is an experimental project exploring training an RWKV-8 Heron model with 1-bit ROSA activation on consumer hardware. It successfully trains a roughly 300 million parameter, 32-layer model within a 1.5GB RAM footprint on a base M1 Mac.
Why it’s interesting
It demonstrates training a multi-hundred-million parameter model within a very low memory footprint on consumer Apple Silicon hardware.
Source metrics: Points 2 · Comments 0
HN discussion
Source: #HackerNews / Show HN
Summary
Ullis is an experimental project exploring training an RWKV-8 Heron model with 1-bit ROSA activation on consumer hardware. It successfully trains a roughly 300 million parameter, 32-layer model within a 1.5GB RAM footprint on a base M1 Mac.
Why it’s interesting
It demonstrates training a multi-hundred-million parameter model within a very low memory footprint on consumer Apple Silicon hardware.
Source metrics: Points 2 · Comments 0
HN discussion
Source: #HackerNews / Show HN