About a decade ago, I tried to index the web via the Dewey Decimal System. I had a site laid out similar to Google, where you could browse sites continuously starting from a given call, but the DDS is proprietary, and those people hate anyone who uses their IP without a license. You can Google everyone they’ve shutdown – places that weren’t even libraries – for using anything similar to the DDS. I reached out to the group that manages the DDS, and was taken offline before my project even started.
With all the corporate BS lately, and people looking for alternative options, I thought I’d take a search engine old school, and we’d index like Usenet. Except with an XXX.XXX.XXX format.
My original project used sharded Redis, with append logs to disk. I chose it for its in-memory speed and key-value store. Did some calculations, and I’d have to have millions of records just to consume my entire system’s memory.
I had a lot of plans for this before getting shut down.
Now that I’m older, I’m curious if I should be using MongoDB.
What are the benefits and drawbacks of each? Which would you use? And why?
I get it. SQL is such an obtuse language. There are no shortage of programming interfaces to it because it is so dumb.
Now, do you hate SQL or RDBMS? Do the concepts of normalization (deduplication of data), primary and foreign keys, etc. bother you? (Not throwing shade, I hate matrix math, we don’t control what we like.) I ask because there are alternatives to not dealing with SQL but maintaining the benefits of a RDBMS - namely data integrity checks, atomic operations, etc. etc.
Because if you do all the programming necessary to guarantee data integrity, and all that good stuff on top of a key-value DB… well, you’ve just created an RDBMS from first principles.
It’s your passion project, use what makes you enjoy it. Just try not to fall into the trap of reinventing the wheel. It’s easier to use an RDBMS for it at first and switching to a more specialized tool later when requirements are clearer than the other way around.
An alternative is finding someone to partner up with and be your DB developer and DBA. Easier said than done, but it’s a great alternative.