Speeding Up Slow Sums – Faster BSD Checksums

I recently made some changes to the Entangle archive that required a good majority of the archive to be checksum reverified. That’s about 200 TB worth of checksums. Even with six systems running parallel, and RAIDs hitting 600 MB/s, it still takes a while.

One thing I noticed was that the I/O rate while the sums were being performed was ~210 MB/s. A respectable speed, but cp’ing a file from the RAID to to /dev/null results in ~600 MB/s. So either my CPU was too slow or the ‘sum’ operation wasn’t as efficient as it could be. Continue reading “Speeding Up Slow Sums – Faster BSD Checksums”

Project Entangle

Project Entangle is Koherence’s internal platform for exploring various media technologies. It began as a test bed for investigating how TiVo-style trick modes could be implemented using HTTP Live Streaming. OTA recording, program guide support, and content search came along shortly after. Deployed Entangle systems are currently being used as DVRs, with users watching shows on iPhones, iPads and AppleTVs. Continue reading “Project Entangle”