Neural Networks Compress Video and Audio into SIREN Weights

Traditional video codecs compress pixel data, but a new approach proposes storing entire video and audio streams as the learned weights of a neural network. Specifically, a sinusoidal representation network (SIREN) is trained to map space-time coordinates to RGB values and audio amplitudes. This network includes separate initialization layers for audio and video, shared hidden layers, and distinct output branches for each modality, with Siamese audio branches used for noise estimation. After training this "teacher" network to overfit the media, its knowledge is distilled into a smaller "student" network. This student network then undergoes 16-bit symmetric weight quantization, followed by lossless LZMA2 encoding for further compression. In tests, this pipeline achieved a 2.61 compression ratio on a 6.08 MiB video, reducing it to 2.33 MiB, with acceptable video and audio quality metrics. While the approach shows promise, it currently falls short of established codecs like H.264, HEVC, and MP3 in certain aspects. A browser-based prototype exists for training, transfer, and decoding these neural representations via WebRTC.