Speeding Up Slow Sums – Faster BSD Checksums

I recently made some changes to the Entangle archive that required a good majority of the archive to be checksum reverified. That’s about 200 TB worth of checksums. Even with six systems running parallel, and RAIDs hitting 600 MB/s, it still takes a while.

One thing I noticed was that the I/O rate while the sums were being performed was ~210 MB/s. A respectable speed, but cp’ing a file from the RAID to to /dev/null results in ~600 MB/s. So either my CPU was too slow or the ‘sum’ operation wasn’t as efficient as it could be. Continue reading “Speeding Up Slow Sums – Faster BSD Checksums”

Project Entangle

Project Entangle is Koherence’s internal platform for developing various media technologies. It began as a test bed for investigating how TiVo-style trick modes could be implemented using HTTP Live Streaming. Since then Project Entangle has blossomed into a collection of platforms.

The Entangle DVR platform supporting in-home and out-of-home playback is currently being alpha tested, with users watching shows on iPhones, iPads and AppleTVs. Another platform variant powers the SFBayATSC ATSC broadcast monitoring system. Continue reading “Project Entangle”