About a decade ago, I tried to index the web via the Dewey Decimal System. I had a site laid out similar to Google, where you could browse sites continuously starting from a given call, but the DDS is proprietary, and those people hate anyone who uses their IP without a license. You can Google everyone they’ve shutdown – places that weren’t even libraries – for using anything similar to the DDS. I reached out to the group that manages the DDS, and was taken offline before my project even started.
With all the corporate BS lately, and people looking for alternative options, I thought I’d take a search engine old school, and we’d index like Usenet. Except with an XXX.XXX.XXX format.
My original project used sharded Redis, with append logs to disk. I chose it for its in-memory speed and key-value store. Did some calculations, and I’d have to have millions of records just to consume my entire system’s memory.
I had a lot of plans for this before getting shut down.
Now that I’m older, I’m curious if I should be using MongoDB.
What are the benefits and drawbacks of each? Which would you use? And why?
Redis is not a database. If you use it like one, it won’t be long before you regret it.
Can I ask why? Are there errors in the append log? Does it forget things?
Not questioning your character. Just curious what your personal experiences were.
Reddis is an in memory cache. Your data will definitely get deleted from Reddis once the amount stored increases or even simply restarting the service. You need proper persistent storage as well so anything that’s not in the cache can still be retrieved, albeit slower. Personally, I think NoSQL is overhyped and would recommend Postgres instead, unless your application has some very specific requirements.
Technically, it shares a lot of characteristics with databases. You can put data in there and get data out again later, sure. But it’s optimized for short term caching of data, to reduce load on a real database, reduce network requests, that kind of thing.
In programming you can often use tools in ways they were not intended for and it’ll still work but the costs of that decision might not become apparent until much later. This is one of those. Over time you’ll start to have issues with durability guarantees, complex querying and indexing, transactions and consistency, recovery after failure, backups and restoration, data growth, operational tooling, migrations and schema evolution, concurrency, debugging and observability. None of that shows up during initial development work so you won’t notice until you’re in way too deep.
So to answer the original question - use Redis AND MongoDB. For different things.
MongoDB is your database and Redis is the thing that stops you from needing to hit up the database all the time.
Depends on the data you want to store and what you want to do with it. I haven’t used either MongoDB or Redis in particular, but MongoDB (or e.g. CouchDB) is a document-oriented database, whereas Redis/Valkey is a key-value store.
Superficially, these are similar kinds of databases, in that you have a key and store a value. But document-oriented typically enforces the format somewhat, like e.g. some JSON format, which is then used for doing indexing, aggregation, search or whatever in the database.
Meanwhile, a key-value store does not care what data is actually stored inside. It can literally be just some bytes. You tell it the key and it tells you the data you previously stored, and that’s all you do with it.
I wouldn’t be surprised, though, if Redis doesn’t fit that definition 100% and actually has some optional document-oriented features as well.From your very short description of what you want to do, my intuition would be more towards document-oriented, because the indexing is presumably essential for a search engine.
But as someone else already said, MongoDB is proprietary now, so probably want to look for an alternative. Selecting the database is worth spending some time on, since the right database might already bring along the search capabilities you want.
I also want to throw in a third type of database that you should probably look into: https://en.wikipedia.org/wiki/Graph_database
Graph databases are good at storing relationships between entries, which you will have a lot of, given each webpage links to other webpages. I believe, the big-boy search engines do use graph databases.
Mongodb is proprietary.
Postgres is open source, and then there is documentdb which adds a mongodb compatible layer on top.
But it would probably be better just to use postgres + jsonb directly for that case.
Mongodb is proprietary.
Actually, MongoDB is webscale.
But I agree, I’d generally recommend a relational database like Postgres, especially for something that needs to be searchable quickly like an index.
Redis is apparently open-source again, now under AGPL (and two other proprietary licenses). But yeah, can also recommend the fork that spawned when they went proprietary: https://en.wikipedia.org/wiki/Valkey
When did Mongo go proprietary?
Edit: I’m actually wanting to make my entire project open source. MongoDB’s current license, requiring all infrastructure to be open, might actually be for me.
No, the license is not a real OSS license.
The short version is that the requirement of all infrastructure being open, could potentially apply to parts of the stack where you don’t have access to the source code, like the management engine or network card firmware.
It hasn’t been tested in court yet, and because this is unclear, the SSPL is a hot potato nobody wants to touch or test. The possibility of the requirement for closed source firmware to be open, makes the SSPL effectively unusable.
The SSPL is neither Open Source Initiative nor Free Software Foundation approved: https://en.wikipedia.org/wiki/Server_Side_Public_License
Currently, SSPL licensed code is best treated as source available, but proprietary code.
Postgres is usually the DB to use unless you have a specific reason to use something else. Something like Cassandra or scylla might be good for your use case but I’d probably still start with postgres and evaluate a switch to something else if and when you need to
I absolutely detest SQL. I will probably go to my deathbed having never typed a line of SQL in my life. I don’t know why I hate it as much as I do, but I refuse to learn it – my brain won’t let me.
I’ve programmed in C, C++, Python, BASIC, JavaScript, Assembly, etc. I wrote CGIs for Apache in all these languages, so I didn’t have to touch PHP. Every time I get near SQL or PHP, I start wanting to pull my hair out.
How can you hate it if you haven’t learned it? You don’t even know what it is.
So this, then: DocumentDB
SQL is certainly a different way of thinking, unlike any other language. I can see why it’s not for everyone.
And not just thinking, it actively imposes an architecture onto your codebase. It pretty much forces a CRUD API, which only really works well for a client-server structure. And it forces you to break up your data structures and introduce IDs to a degree that you would simply not do while programming normally.
I mean, I can see the appeal. If these constraints are fine for you and you don’t have other constraints, like sparse data, then building your whole application on top of SQL gives you clear answers for how to architect that.
But yeah, I also really resent this idea that it should be the default, because when it does not match the problem domain, you spend a lot of time working around the architecture that it imposes.
You can go very minimal on the sql in postgres if you choose. You can mimic mongo’s basic function using the jsonb columns. So you’ll still be inserting data. But that will be indexed json. Here’s a docs link. Oh and I also vote mongo
Oh, so you’re insane
You could say that.
I wrote my own key-value database (with drive and memory storage), so I didn’t have to use SQL.
https://github.com/ki4jgt/PPD/
Just got bored one night and built a blog off of it.
Edit: I’m literally tempted to write an API server around this, add sharding, and call it a night.
