Storage and Data Hoarding

I am a hoarder, not in the clinical sense but in the digital sense. Well also a little in the physical sense too when it comes to books and stuff. I guess I hoard information, I don’t know why but there is a part of me that fears losing any data and that I have to do my part to preserve it as best I can.

I could go on about my physical hoarding (just books… and computer parts… and magazines….) but that’s probably more of a me problem than anything else. No, what I want to talk about is data hoarding. I have a lot of data, a mixed bag of years of information. Lately, with my film project, the data has started to grow exponentially. Raw footage takes up a lot of space and I can’t bear to delete it when I have uploaded the final footage to YouTube. So what’s a guy to do?

My storage needs have recently ballooned from five terabytes to ten and that ten is if I am being conservative. Realistically I could use 20 or more (ideally more). A full 5 minute short (that I make daily) takes around 10 to 15 gb of space. This is on the low end, I recently stopped shooting in interlaced video (due to the headache it was causing with editing). Those shorts would be significantly larger.

Since those are daily you can see the storage growth. That’s about a terabyte worth of storage required every sixty six days. I’m not deleting any older stuff either so that will only grow, and even more so as I move into longer form content. So herein lies the dilemma.

I need to keep that data safe, along with my other non video data (programming, writing etc.). In order for data to be truly safe it needs to be in at least three places. One “hot” storage (the copy on your hard drive), one backup that is ideally cold (a backup on a removable for example) and one offsite. This keeps everything safe from both drive failure and potential disaster (fire, flood, theft etc.).

That means for ten terabytes of storage I need thirty terabytes (actually forty because RAID). That’s a lot of cash, especially for the offsite stuff as there are monthly fees associated with that. That is where my dilemma lies, what is the most cost effective way to accomplish this?

Well, for my hot storage I already own all the drives. My onsite duplication is most likely going to be a NAS. This will centralize all my data. The ten terabytes I need is actually spread over a couple of machines (laptop, gaming rig, workstation etc.). Having an onsite central place for it all will be nice.

Now, that requires a whole build and a bunch of other stuff (aka money) but it’s not too crazy. I have a plan there. Getting that offsite is the next step. I’m thinking of either setting up a process where it automatically backs up to Wasabi Cloud ( a low cost Amazon S3 storage cloud) or somehow getting it to work with Backblaze’s unlimited backup plan (which would be cheaper). That will take a lot of thinking though, probably will have to try a few things out.

Now, I know my system has some weaknesses. I really should implement a cold storage back up in there because a virus could wipe most of my home stuff out and potentially impact the offsite stuff. Same with a corrupt file or accidental deletion. I think I’ll be okay because there is only one user (me).

Most likely I will do a weekly manual backup to a single drive that I plug in to the NAS once a week for that purpose. The problem there is time / remembering to do it. Human error is the biggest risk in all honesty.

The level to which I have thought about this problem I think shows how horading data is kind of like hoarding physical things. It takes over your life. Oh well, it’s fun and handy.

Now just to get the money together for all this stuff.

Kind of a rant but I figured I would share, it’s a little techy and nerdy but it’s another thing I enjoy doing. And in today’s digital world it is a good thing to set up. Honestly, everyone should be backing up their data at least to the cloud. Data is gold these days, we should probably treat it as such. It used to be you would only lose all your important information in the event of a disaster, now a simple drive failure or dropped laptop can mean losing years of valuable information. Things like photographs, tax documents, legal documents and other irreplaceable stuff. Long story short, even if your needs aren’t as heavy as mine you still need to back up your data. 

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 537 other subscribers

Leave a Reply