/j hey some of us only have 10GbE
Jokes aside, I get the classification. I’m pretty solidly in the category of 2c - more tech than some medium business but without the SLA to go with it.
/j hey some of us only have 10GbE
Jokes aside, I get the classification. I’m pretty solidly in the category of 2c - more tech than some medium business but without the SLA to go with it.
Apologies for being late, I wanted to be as correct as I could be.
So, straight to the point: Nextcloud by default uses plain files if you don’t configure the primary storage to be an S3/object store. As far as I can tell, this is not automatic and is an intentional change at system creation by the original admin. There is a third-party migration script, but there does not appear to be a first-party method of converting between the two. That’s very good news for you! (I think/hope)
My instance was set up as a standalone, so I cannot speak for the all-in-one image. Poking around the root data directory (datadirectory
in the config.php
), I was able to locate my user account by internal username - which if you do not use LDAP will be the shortened login name. On default LDAP configs, this internal username may be a GUID, but that can be changed during the LDAP enablement process by overriding the Internal Username field in the Expert LDAP settings.
Once in the user’s home folder in the root data directory, my subdirectory options are cache, files, files_trashbin, files_versions, uploads.
files
contains the “live” structure of how I perceive my Nextcloud home folder in the Web UI and the Nextcloud Desktop sync enginefiles_trashbin
is an unstructured data folder containing every file that was deleted by this user and kept per the trash folder’s retention policy (this can be configured at the site level). Files retain their original name, but have a suffix added which takes the form .d######...
where the numbers appear to be a Unix timestamp, likely the deletion date. A quick scan of these with the file
command in Linux showed that each one had an expected file header based on its extension (i.e., a .png
showed as a PNG image with an expected resolution). In the Web UI, there is metadata about which folder the file originally resided in, but I was not able to quickly identify this in the file structure. I believe this info is coming in from the SQL database.files_version
are how Nextcloud is storing its file version history (if enabled). Old versions are cleaned up per a set of default behaviors to keep more copies of more recent changes, up to a maximum age deletion threshold set at the site level. This folder is stored in approximately the same structure as the main files
live structure, however each copy of each version is appended a suffix .v######...
where the number appears to be the Unix timestamp the version was taken (*I have not verified that this exactly matches what the UI shows, nor have I read the source code that generates this). I’ve spot checked via the Linux file
command and sha256 that the files in this versions structure appear to be real data - tested one Excel doc and one plain text doc.I think that should get a fairly rough answer to your original question, but if I left something out you’re curious about, let me know.
Finally, I wanted to thank you for making me actually take a look at how I had decided to configure and back up my Nextcloud instance and ngl it was kind of a mess. The trash bin and versions can both get out of hand if you have frequently changing or deleting/recreating files (I have network synchronization glued onto some of my games that do not have good remote save support). Retention policy on trash and versions cleaned up extraneous data a lot, as only one of those was partially configured.
I can see a lot of room for improvements… just gotta rip the band-aid off and make intelligent decisions rather than just slapping an rsync
job that connects to the Nextcloud instance and replicates down the files and backend database. Not terrible, but not great.
In the backend I’m already using ZFS for my files and Redis database, but my core SQL database was located on the server’s root partition (which is XFS - I’d rather not mess with a DKMS module from a boot CD if something happens and upstream borks the compile, which is precisely what happened when I upgraded to OpenZFS 2.1.15).
I do not have automatic ZFS snapshots configured at this time, but based on the above, I’m reasonably confident that I could get data back from a ZFS snapshot if any of the normal guardrails within Nextcloud failed or did not work as intended (trash bin and internal version history). Plus, the data in that cursed rsync backup should be at least 90% functional.
I forget where I originally found this and Google on my phone was unhelpful.
My favorite annoying trick is x -=- 1
. It looks like it shouldn’t work because that’s not precisely a valid operator, but it functions as an increment equivalent to x += 1
It works because -=
still functions are “subtract and assign”, but the second minus applies to the 1 making it -1.
Probably best to go with something in the 3.5" line, unless you’re going enterprise 2.5" (which are entirely different birds than consumer drives)
Whatever you get for your NAS, make sure it’s CMR and not SMR. SMR drives do not perform well in NAS arrays.
Many years ago I for some low cost 2.5" Barracuda for my servers only to find out years after I bought them that they were SMR and that may have been a contributing factor to them not being as fast as I expected.
TLDR: Read the datasheet
I don’t have a full answer to snapshots right now, but I can confirm Nextcloud has VFS support on Windows. I’ve been working on a project to move myself over to it from Syno drive. Client wise, the two have fairly similar features with one exception - Nextcloud generates one Explorer sidebar object per connection, which I think Synology handles as shortcuts in the one directory. If prefer if NC did the later or allowed me to choose, but I’m happier with what I got for now.
As for the snapshotting, you should be able to snapshot the underlying FS/DB at the same time, but I haven’t poked deeply at that. Files I believe are plain (I will disassemble my nextcloud server to confirm this tonight and update my comment), but some do preserve version history so I want to be sure before I give you final confirmation. The Nextcloud root data directory is broken up by internal user ID, which is an immutable field (you cannot change your username even in LDAP), probably because of this filesystem.
One thing that may interest you is the external storage feature, which I’ve been working on migrating a large data set I have to:
Admin docs for reference: https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage_configuration_gui.html
I use LDAP user auth to my nextcloud, with two external shares to my NAS using a pass-through session password (the NAS is AD joined to the same domain as Nextcloud uses for LDAPS). I don’t know if/how the “store password in database” option is encrypted, but if anyone knows I would be curious, because using session passwords prevents the user from sharing the folder to at least a federated destination (I tried with my friend’s NC server, haven’t tried with a local user yet but I assume the same limitations apply). If that’s your vibe, then this is a feature XD.
One of my two external storage mounts is a “common” share with multiple users accessing the same directory, and the second share is \\nas.example.com\home\nextcloud. Internally, these I believe is handled by PHP spawning smbclient
subprocesses, so if you have lots of remote files and don’t want to nuke your Nextcloud, you will probably need to increase the PHP child limits (that too me too long to solve lol)
That funny sub-mount name above handles an edge case where Nextcloud/DAV can’t handle directories with certain characters - notably the # that Synology uses to expose their #recycle and #snapshot structures. This means that remote mount to SMB has a limitation at the moment where you can’t mount the base share of a Synology NAS that has this feature enabled. I tried a server-side Nextcloud plugin to try to filter this out before it exposed to DAV, but it was glitchy. Unsure if this was because I just had too many files for it to handle thanks to the way Synology snapshots are exposed or if it actually was something else - either way I worked around the problem for now by not ever mounting a base share of my Synology NAS. Other snapshot exposure methods may be affected - I have a ZFS TrueNAS Core, so maybe I’ll throw that at it and see if I can break Nextcloud again :P
Edit addon: OP just so I answer your real question when I get to this again this evening - when you said that Nextcloud might not meet your needs, was your concern specifically the server-side data format? I assume from the rest of your questions that you’re concerned with data resilience and the ability to get your data back without any vendor tools - that it will just be there when you need it.
Fair - there are ways to handle it. I didn’t want to include specifics since I’m not a professional contractor for this sort of thing, but I should have indicated that there are exceptions.
Full extension rails are probably best going to come from the original vendor as a general principle, rather than attempting to use universal rails.
If you have a wall mounted rack, unless your walls are not drywall, physics is working against you. It’s already a pretty intense heavy cantilever, and putting a server in there that can extend past the front edge is only going to make that worse.
If you want to use full extension rails, you should get a rack that can sit squarely on the floor on either feet or appropriately rated casters. You should also make sure your heaviest items are on the bottom ESPECIALLY if you have full extension rails - it will make the rack less likely to overbalance itself and tip over when the server is extended.
I’m not entirely sure what you’re seeking to accomplish here - are you looking to just impose authorization on a subset of the images? Probably those should be in a non-public bucket for starters.
Looking to only give certain people access to files and also have a nicer UI (a la Google Drive / Photos)? Maybe plain S3 isn’t the play here and a dedicated application is needed for that subset.
Pre signed URLs may also be a thing useful to what you’re trying to to solve. https://docs.min.io/docs/javascript-client-api-reference.html#presignedGetObject
Adding on one aspect to things others have mentioned here.
I personally have both ports/URLs opened and VPN-only services.
IMHO, it also depends on the exposure tolerance the software has or risk of what could get compromised if an attacker were to find the password.
Start by thinking of the VPN itself (Taliscale, Wireguard, OpenVPN, IPSec/IKEv2, Zerotier) as a service just like the service your considering exposing.
Almost all (working on the all part lol) of my external services require TOTP/2FA and are required to be directly exposed - i.e. VPN gateway, jump host, file server (nextcloud), git server, PBX, music reflector I used for D&D, game servers shared with friends. Those ones I either absolutely need to be external (VPN, jump) or are external so that I don’t have to deal with the complicated networking of per-user firewalls so my friends don’t need to VPN to me to get something done.
The second part for me is tolerance to be external and what risk it is if it got popped. I have a LOT of things I just don’t want on the web - my VM control panels (proxmox, vSphere, XCP), my UPS/PDU, my NAS control panel, my monitoring server, my SMB/RDP sessions, etc. That kind of stuff is super high risk - there’s a lot of damage that someone could do with that, a LOT of attack surface area, and, especially in the case of embedded firmware like the UPSs and PDUs, potentially software that the vendor hasn’t updated in years with who-knows-what bugs lurking in it.
So there’s not really a one size fits all kind of situation. You have to address the needs of each service you host on a case by case basis. Some potential questions to ask yourself (but obviously a non-exhaustive list):
So, as you can see, it’s not just cut and dry. You have to think about each service you host and what it does.
Larger well known products - such as Guacamole, Nextcloud, Owncloud, strongswan, OpenVPN, Wireguard - are known to behave well under these circumstances. That’s going to factor in to this too. Many times the right answer will be to expose a port - the most important thing is to make an active decision to do so.
I’m not the commenter but I can take a guess - I would assume “data source” refers to a machine readable database or aggregator.
Making the system capable of turning off a generic external service in an automated way isn’t necessarily trivial, but it’s doable given appropriate systems.
Knowing when to turn a service off is going to be the million dollar question. It not only has to determine what the backend application version is during its periodic health check, it also needs to then make an autonomous decision that a vulnerability exists and is severe enough to take action.
Home Assistant probably provides a “safe list” of versions that instances regularly pull down and automatically disconnect if they determine themselves to be affected, or, of the remote UI connection passes through the Home Assistant Central servers, the Central servers could maintain that safety database and off switch. (Note - I don’t have a home assistant so I can’t check myself)
Somewhat halfway between practical use and just messing around for fun.
Several years ago I built a GPS NTP clock out of an RPi3 and an Adafruit GPS hat. Once I had the PPS driver installed, it’s precision/drift got pretty good. According to its own self measurements, I got pretty dang close to NIST stratum 1 NTP servers, but those are hundreds of miles away so that measurement isn’t super precise. It’s still running today, clocking nearly 24/7 operation since (checks shopping history) 2017, though I replaced the breadboard and mini module with a full sized hat with the same chipset in 2021.
Recently I acquired a proper hardware GPS clock and I stacked the two against each other and found out my RPi did not half bad and can get between 0.5-10ms of the professionals (literally I’m pretty sure I’d need more precise measuring equipment to tell the difference between the two at this point than a regular computer). Now my homelab has fully redundant internet-disconneted stratum 1 time. Been half considering if I could write a GPSD driver for it as a joke, but I know upstream won’t accept it because it doesn’t offer SOOO many features they’d need.
As for what else - I just kind of keep an eye out for projects related to GPS and high precision time, like the open source atomic PCI card that was released a few years ago. Finding out what people are doing to get better and better time is just downright interesting.
Outside of the time world, it’s just fun to see what projects people come up with relating to maps and navigation. Stretch goal once I have enough server horsepower is to make a render-capable Open Street Map server with my home region loaded to start with, but eventually I’d like to get it to the point where I can load and process world.osm. That… Requires a LOT of CPU and SSD space.
Heyo, just wanted to say I appreciate the edit.
Some people see three extra clicks (which is what it took on mobile to get the real description out of GitHub) as a limiter. I actually clicked because I had guessed that with a name like “navidrome” it was something GNSS related, was surprised to see it was about music.
I’ve been self hosting for going on 7-8y, following various communities on reddit and Lemmy and I learn about new softwares every day. I’ll have to toss this one on my investigation queue.
Other have all mentioned various tech to help you out with this - Ceph, ZFS, RAID at 50/60, raid is not a backup, etc.
40 drives is a massive amount. On my system, I have ~58TB (before filesystem overhead) comprised of a 48TB NAS (5x12TB@RAID-5) 42TB of USB backup disks for said NAS (RAID is not a backup), a 3-node vSAN array with 12TB (3x500GB cache, 6x2TB capacity) of all-flash storage at RF2 (so ~6TB usable, since each VM is in independent RAID-1), and a standalone host with ~4TB@RAID-5 (16 disks spread across 2 RAID-5 arrays, don’t have the numbers off hand)
That’s 5+9+16=30 drives, and the whole rack takes up 950w including the switches, which iirc account for ~250-300w (I need to upgrade those to non-PoE versions to save on some juice). Each server on its own takes up 112-185w, as measured at iDRAC. It used to take up 1100w until I upgraded some of my older servers into newer ones with better power efficiency as my own build-out design principle.
While you can just throw 40-60 drives in a 4u chassis (both Dell and 45drives/Storinator offer the ability to have this as a DAS or server), that thing will be GIGA heavy fully loaded. Make sure you have enough power (my rack has a dedicated circuit) and that you place the rack on a stable floor surface capable of withstanding hundreds of pounds on four wheels (I think I estimated my rack to be in the 300-500lbs class)
You mentioned wanting to watch videos for knowledge - if you want anywhere to start, I’d like you to start by watching the series Linus Tech Tips did on their Petabyte Project’s many iterations as a case study for understanding what can go wrong when you have that many drives. Then look into the tech you can use to not make the same mistakes Linus did. Many very very good options are discussed in the other comments here, and I’ve already rambled on far too long.
Other than that, I wish you the best of luck on your NAS journey, friend. Running this stuff can be fun and challenging, but in the end you have a sweet system if you pull it off. :) There’s just a few hurdles to cross since at the 140TB size, you’re basically in enterprise storage land with all the fun problems that come with scale up/out. May your storage be plentiful and your drives stay spinning.
It is far less improbable than you think, especially if all of your drives have similar age/wear - as would be the case if you bought all 40 around the same time.
I don’t think I’ve seen that movie, so that explains why I missed the joke
I’m still lost… I’ve been following the XZ thing since it broke, so I get the context, but I’m not sure how the meme at the bottom is connected?
Isn’t the point of PGP/GPG that there’s no central database?
As others said, depends on your use case. There are lots of good discussions here about mirroring vs single disks, different vendors, etc. Some backup systems may want you to have a large filesystem available that would not be otherwise attainable without a RAID 5/6.
Enterprise backups tend to fall along the recommendation called 3-2-1:
On my home system, I have 3-2-0 for most data and 4-3-0 for my most important virtual machines. My home system doesn’t have an off-site, but I do have two external hard drives connected to my NAS.
Story time
I had one of my two backup drives fail a few months ago. Literally actually nothing of value was lost, just went down to the electronics shop and bought a bigger drive from the same vendor (preserving the one on each vendor approach). Reformatted the disk, recreated the backup job, then ran the first transfer. Pretty much not a big deal, all the data was still in 2 other places - the source itself, and the NAS primary array.
The most important thing to determine about a backup when you plan one - think about how much the data is valuable to you. That’s how much you might be willing to spend on keeping that data safe.
Running nextcloud (non docker version) and I don’t see near so many client updates - usually once every few weeks, which would be a reasonable expected pace. Server updates are less frequent.
On Windows (all of my primary devices), I just install the NC client update and skip the explorer restart, pending full reboot later. Tis the nature of literally anything that deeply integrates with Explorer. I’ve seen explorer “death” during updates from several vendors that have similar explorer plugins, not just NC. Explorer sometimes just decides to nope out even without NC updating.
Now on one device I hadn’t opened for a while, I saw NC run two updates in a row, but that was my fault for procrastinating the first one.
Here’s the desktop release history: https://github.com/nextcloud/desktop/releases
I don’t see a “one every day” within the block of time between Dec 6 and today, unless you had the release candidate builds which may have been more frequent in a few spots.
Having used Dell BOSS S1 cards even in other Dell servers, there is firmware integration limitations to be seen. Even an R730 won’t fully tolerate an R740’s BOSS - those I’ve only seen work in the R740 and higher. The control interface for the S1 (and presumably S2) cards is integrated into a menu system in Dell’s BIOS and iDRAC.
I know specifically for the BOSS-S1, there’s a Startech board that has a similar form factor and uses the same SATA RAID chipset, but without the Dell firmware. That Startech board works in non-Dell servers and workstations and has mostly the same features as the S1. (I 100% used my laptop’s eGPU to set them up a few times. It also definitely causes Windows to BSOD a few times because it doesn’t know how to eject an entire disk controller, but that’s also entirely my fault). The Startech controller has not really given me any major problems once I got it up, and has run in my R730s with near 100% uptime for a few months now.
You may look to see if Startech has an NVMe version now to find a counterpart for the BOSS-N1 - I haven’t checked recently.
Something else to consider is what your RAID array will actually be doing: M.2 SATA may be fast enough to be a boot disk, while your “real” data array uses SAS or NVMe to get to the CPU. You can even elect to use something fancier like Ceph or ZFS to handle the real data disks without a hardware RAID card. If you’re just booting the server hypervisor and maybe a low level agent VM or two and the real data is on another array, that Startech card may be for you. (You just need FreeDOS to re flash it to EFI mode)