Hopefully this is my final edit before getting a full solution but I just want to say that if you are going to either resort to belittling me for not using better/newer hardware or you make it obvious that you haven’t fully read the post before commenting, I’m not going to respond to your comment.

In Windows, there’s a file/folder option called “Compress contents to save disk space”. What it does is it compresses the files, as the name suggests, but leaves them accessible as though they aren’t. This doesn’t really have much of a benefit on newer storage devices but on older storage devices, in addition to saving space, it allows files to potentially read faster.

As I have some old storage devices that I want to run games from, I think this will be a great option to have if I could find something similar for Linux. I tried looking online myself but search engines are terrible and I couldn’t find anything though them. So, I decided to post about this here, to see if anyone knows of anything I could try.

Edit: I have figured out how to use BTRFS and enable what it calls “transparent file compression”. Games are running decently well and I’m able to run games that are much larger than the devices original capacity (it seems to be around 2 to 2.5 the original free space, on average), so I’m going to use that on most of my old storage devices at least for the time being. The only problem I’m having is that I want to use F2FS on my oldest storage device, as BTRFS takes up too much space on that device.

When formatted to BTRFS, there’s only about 40MB of free space and with compression it can hold around 100MB worth of files. Meanwhile, there’s about 80MB of free space with F2FS and I was told by multiple users that F2FS also supports transparent file compression, which should get me around 200MB worth of files on that device because it’s documented to support the same compression methods as BTRFS. But I can’t get files to compress and I’m not getting any error messages to try and diagnose the problem. Based on what the documentation says, I’m supposed to do something like this:

sudo mkfs.f2fs -f -O extra_attr,inode_checksum,sb_checksum,compression /dev/mmcblk0p1
sudo mount -o compress_algorithm=zstd,compress_extension=* /dev/mmcblk0p1 '/home/j/mountpoint/128mb'
chattr -R +c '/home/j/mountpoint/128mb'

The device will mount like this but files aren’t compressing when added, nor are they compressed if using the last command after they’ve been moved.

  • tiptoes@sh.itjust.works
    19·
    6 months ago

    The equivalent would be either zfs or btrfs compression. Transparent to applications, you don’t have to do anything special other than enable it.

      • BananaTrifleViolin@piefed.worldEnglish
        6·
        6 months ago

        How small is your smallest device? BTRFS doesn’t have a minimum size, but practically probably 50-100mb is just about doable before even just setting things up get complex. Having said that though, it’s copy-on-write and has overhead as a result, so may not function well below 1gb.

        ZFS meanwhile really won’t work well below probably 8gb. It’s also copy-on-write but with a lot more overhead due to how it works. It really works best on big drives and filesystems.

        If your old storage is in the mb range, then really neither will help you achieve what you want.

        BTRFS and ZFS do offer the same benefits as NTFS with regard to compression and speeding up some slower devices (due to lowering the actual read/writes needed to achieve the same result as the data is compressed into a smaller space and decompressed rapidly by the PC in memory), but NTFS can go be used on much smaller disk sizes due to how it works. BTRFS and ZFS are designed and optimised with other benefits in mind. And NTFS compression isn’t well supported in Linux.

      • tiptoes@sh.itjust.works
        6·
        6 months ago

        How small are these devices? I think the other problem is that neither BTRFS nor zfs really are suitable for removable devices, and definitely not for ones smaller than probably 8Gb at the very least.

        Unlike NTFS which is just a file system, both BTRFS and ZFS do volume management too, so it’s not just a single partition thing; they prefer to take over an entire volume and manage everything.

        So while they’re the closest filesystem with NTFS-like transparent compression……they don’t match exactly.

        I also hazard to guess if the devices you’re using are too small to accept a BTRFS formatted volume, no amount of compression is going to be enough to fit what you need.

        If you just want to play with a bunch of small old devices……maybe play with LVM and small RAID arrays and configurations instead. You can the build a bigger volume out of a bunch of those disks together and then put a BTRFS or zfs volume on them. Can be fun to experiment and learn with anyway.

          • ivn@tarte.nuage-libre.frFrançais
            2·
            6 months ago

            You’ll need to read the documentation to understand its concepts first. You don’t use mkfs but zpool to create a pool then use zfs to create the fs on it. ZFS is usually used with some sort of raid configuration so most doc will show this but you can create a pool with no replication using a single drive.

  • Shimitar@downonthestreet.euEnglish
    4·
    6 months ago

    This is at file system level… Checkout btrfs and zfs, I am quite positive both can compress like you want.

    Never used this feature myself, so cannot be more specific.

    Also, there are some read-only compressed filesystems for Linux that you can also use, they offer best compression but data is read only.

      • Shimitar@downonthestreet.euEnglish
        3·
        6 months ago

        How small is that device??? Didn’t even knew btrfs has a minimum size.

        After for zfs, you probably have to install some software or kernel patches

          • edinbruh@feddit.itEnglish
            3·
            6 months ago

            If it’s a flash memory (sd card, usb stick, ssd, etc), you could try f2fs, it’s very light, and it supports compression and is meant specifically for that kind of devices (well, more for ssds).

            But judging your experience from your comments, I suggest you don’t delve into niche filesystems until you have more experience with Linux, especially for something like 128MB. I especially suggest you avoid zfs for now.

              • edinbruh@feddit.itEnglish
                0·
                6 months ago

                So, first of all, there is no gui for this, that I’m aware of, so you will have to do it from terminal. Second, on f2fs, compression works that you don’t enable compression for a folder, instead you mount the drive with compression enabled, and new files will be compressed automatically.

                So what you need is to set up your disk to be mounted with compression. There are many paths you can follow here. If you want your drive to be (almost) permanently connected, the easiest way is to use “/etc/fstab”. If you want to use it as a regular SD card, mounting and ejecting it from your file explorer etcetera, then you should go here and learn how to have udisks2 mount your device with compression, which should be what your desktop environment uses to mount drives. I suggest you set that up for your specific device, and not for all f2fs devices. Good luck.

                You can look up other useful f2fs options on the arch wiki. I suggest you add all those options that reduces writes to your disk and improve durability (like lazytime).

                You should use zstd as compression algorithm, and because this is a slow and small drive, you can crank up the level of compression.

                If you manage to pull this off, the next time you install a (bigger and faster) drive on your pc, you can try to look into zfs.

          • hendrik@palaver.p3x.deEnglish
            2·
            6 months ago

            Huh. My computer allows me to format a 128MB image file with brtfs. It won’t do it at 64MB though.

              • hendrik@palaver.p3x.deEnglish
                1·
                6 months ago

                Good call. Seems minimum for ZFS is three quarters of a Gigabyte, anyway. And definitely not made for what OP does 😆

              • hendrik@palaver.p3x.deEnglish
                3·
                6 months ago
                truncate -s 128M fs.img  
parted fs.img  
  mklabel gpt  
  mkpart primary btrfs 1MiB 100%  
  quit  
sudo losetup --find --partscan --show fs.img  
sudo mkfs.btrfs /dev/loop0p1

                You should be able to skip the loop device stuff and work on an actual device instead. Seems to me the limit is somewhere between 64M and 128M.

                Edit: But as edinbruh said, maybe try f2fs if it’s a flash device, that’s probably a bit more lightweight?! And since I don’t know what you’re doing… If it’s embedded stuff and you’re alright with read-only, you might want to use squashfs.

  • wltr@discuss.tchncs.de
    3·
    6 months ago

    Not sure that’s relevant, but I’m playing around with Linux on some obsolete Windows tablets with cheap eMMC disks (one of which is broken, so I replaced it with a no name microSD card, plus USB drive), and I format my disks to f2fs, which theoretically should help with both keeping the disks for longer, and accessing the data faster.

      • Natanael@slrpnk.net
        4·
        6 months ago

        Make sure you’re actually filling the volume, and also keep in mind reporting may be different (with or without filesystem index metadata, etc)

        Also, you can simply use regular file systems and compressed files, and then use a RAM drive (assuming you have enough RAM free) and access the files that way instead

  • tychosmoose@piefed.socialEnglish
    3·
    6 months ago

    What are you saving on that drive? Many data file formats already have compression of their own and don’t benefit much from file system compression. So if this is for media files, for example, it’s likely to add CPU overhead without a big benefit in transfer speed.

    ZFS is not installed by default with most Linux distributions due to its license. It’s something you install after the os. Btrfs should work, but I see some discussion online of 128 or 256MB minimum volume size.

  • ThyTTY@lemmy.world
    3·
    6 months ago

    There’s an interesting project called DwarFS. I have it on my todolist to check it out but as I understand it you basically create a compressed read-only archive that is mountable and readable just like any other disk - https://github.com/mhx/dwarfs . Maybe this is something for your use case?

  • MonkderVierte@lemmy.zip
    2·
    6 months ago

    What do you mean, btrfs “takes too much space”? That’s not a sentence that belongs to file system stuff in any way.
    The disk is the space. You make it usable with a fs, for which you can partition portions of it.

    And also, i’ve had bad experiences with f2fs long-term stability.

          • Hupf@feddit.org
            1·
            6 months ago

            Mind you that “free space” figures with compression, with btrfs and with such tiny file systems are tricky at best. You’d have to test all options by actually copying your data set first.

            • vortexal@lemmy.mlOP
              1·
              6 months ago

              Yeah, I don’t want to waste my time trying other options. As I said before, it’s working fine the way it is. There’s around 120MB of free space and I was able to fit a little under 250MB worth of files with the folder I used for testing, so this seems pretty as good as is.

              Also, this part isn’t directed at you specifically, but I’m thinking about just deleting my post. It seems like a troll is trying to use my post to effectively harass me by both misinterpreting some lines in my original post and using some others to belittle me. Because I don’t want to deal with them and possible others, I think it’s best I just delete my post. I may also step away from this site for a bit, I dealt with this kinda shit on Reddit and I don’t want to deal with it here.

          • Hupf@feddit.org
            1·
            6 months ago

            You may have to clear it first.

            Check dmesg output for usually more verbose mount errors.

            Also I’d use compress-force and test out the other compression algorithms with your data set.

  • doodoo_wizard@lemmy.ml
    1·
    6 months ago

    My “I don’t need to save space, I want it for a different purpose” tee shirt is raising a lot of questions answered by my “I don’t need to save space, I want it for a different purpose” tee shirt.

  • monovergent@lemmy.ml
    1·
    6 months ago

    F2FS seems to do what you want, it’ll reserve the original size of your file but compress what’s actually written. Performance numbers might be massively inflated if your writes don’t saturate the cache in RAM.

    I’ve used BTRFS on SD cards before and it’s mostly fine, but it will struggle massively if over 90% full, or if you have a < 1 GB volume and are, say, frequently updating a handful of files that together take up more than half its capacity. Mostly due to the CoW mechanism, it needs some headroom to make a copy of whatever files are being modified.

  • ZkhqrD5o@lemmy.world
    1·
    6 months ago

    sudo nano /etc/fstab

    Then replace “defaults” with “compress=zstd” on your desired partitions.

    IMO stick with btrfs. Also, the storage space between file systems is exactly the same, given the same hardware.

      • ZkhqrD5o@lemmy.world
        0·
        6 months ago

        The fstab file is used to define how disk partitions or remote file systems are mounted into your computer. Removable drives such as USB drives or SD cards are not shown there, because if they were, your system would complain at boot that it can’t find the requested USB drive.

        About F2FS having double the storage space of btrfs. In all honesty, it doesn’t make sense to me. Do you mean it shows you potentially being able to store 80 megabytes if you can get compression working or does it just show 80 megabytes instead of 40, after formatting the file system?

      • non_burglar@lemmy.world
        1·
        6 months ago

        BTRFS and ZFS both use block compression, ZFS by default. It’s meant to increase both storage efficiency and access speed and has nearly zero impact on performance. The files aren’t compressed from the filesystem point of view, which would satisfy your requirement not to need any other tools.

        I’m confused about your statement that you don’t want to save space with compression, but you indicated that you want to “make the most of your storage”. Are you looking for long-term archiving?