Show HN: I built open source file sharing solution using AWS S3
s3-file-share-for-free-35n2u.kinsta.appI created a 100% Open source Company-wide Self-hosted File Sharing Solution for Teams
Recently, I wanted to share HD images and video files with my graphic designer. She’s exceptional at her craft but isn’t familiar with AWS S3
So, I got an idea and built this.
Github Repo: https://github.com/rohitg00/s3-file-share-for-free
Detailed Guide: https://ghumare64.medium.com/i-built-a-company-wide-self-hos...
I'm pretty leery of making the "access key" and "secret key" so public (like typing them into a web page, or setting them in environment variables). Of course it adds significant friction to set up an IAM identity for every user, and "low friction" is one of the key requirements here.
A "correct" implementation would give you a temporary IAM role or something (STS) based on a JWT or other authn mechanism.
This is not that difficult if you're already invested in an identity ecosystem, but a right pain without something to bootstrap it.
On the plus side, AWS creds can be made to be temporary and limited in scope to just the nouns/verbs required. Creating and vending those tokens is an exercise for the reader.
It really isn't that challenging to get going with JWT auth in AWS. Gitlab has pretty good documentation for how to use Gitlab ID tokens to assume roles that includes everything other than how to generate a JWT here: https://docs.gitlab.com/ee/ci/cloud_services/aws/
And of course generating OIDC PKI JWTs is pretty easy and well documented elsewhere.
The harder parts in my mind are:
I completely Agree on this point. I have this in mind for implementation. For now, I'm focusing on bringing more cloud providers.
might be fine for internal company use but pasting access/secret key on a third party website will get you a call from security...or worse, won't
My initial though. At least create an IAM user per file :-). Maybe that defeats the convenience.
Cloning and installing is also an option.
I can add this option with just a few changes in a code, but how can it be easy to use for any team?
I am not an IAM expert but maybe the app should have an admin login that sets the IAM user with full permissions on any s3 bucket(s) needed for the app to work.
There should be instructions on how to set that IAM user up (dont make it the root! It just needs full access to a single bucket ideally).
Magic Wormhole gives you secure file transmission for free.
Give one of the implementations a try: https://github.com/psanford/wormhole-william
Why not S3 presigned URLs? It's already baked into the service anyways.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareO...
I agree with other posters that long lived non-autorotating IAM/S3 secret keys are not a good idea. A common alternate approach is presigned URLs. And not just on S3.
Google Cloud, Backblaze, Digital Ocean, Cloudflare, Azure all have this presigned URLs functionality too (I checked for the degree of lockin before I started using S3's presigned URLs in a set of bulk-data APIs at one place I've worked.)
GCP: https://cloud.google.com/storage/docs/access-control/signed-...
Digital Ocean: https://docs.digitalocean.com/products/spaces/how-to/set-fil...
Hetzner: https://docs.hetzner.com/storage/object-storage/faq/buckets-...
Backblaze: https://backblaze-prod.us.document360.io/apidocs/b2-get-down...
Cloudflare R2: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
Azure: https://learn.microsoft.com/en-us/rest/api/storageservices/d...
etc.
Also, presigned URLs can be used not just for downloading files with a temporary URL but can also be created for uploading files with a temporary URL.
Magic wormhole is great for live, peer-to-peer transfers.
But it is not great if you want to distribute a file multiple times, asynchronously, or with other functionality gained from centralized storage. This is where people typically use email, dropbox, or perhaps the tool from TFA.
I think you can seed torrents via HTTP but you'd want to add a layer of file encryption in case someone discovers your torrent
Non-free Resilio Sync basically automated this
You guys waste too much time building “open” stuff on S3.
This is what I was wondering ... why start with s3? Why not "simply" build on IPFS or something like that as a start. Even a shared syncthing key would be MVP.
Quick Update: I didn't expect so many people to show interest in this project. As a few suggested, I plan to add more storage solutions in the next few days.
Testing Other Cloud Providers:
Backblaze B2 ($5/TB/month)
Wasabi ($6/TB/month)
Google Cloud Storage ($20/TB/month) - I am ready to deploy to production.
DigitalOcean Spaces ($5/TB/month)
Cloudflare R2 ($15/TB/month)
Hetzner Storage ($3/TB/month)
This seems like such a horrible, insecure idea that would never pass muster at any company I’ve ever worked at
If you have a dedicated bucket just for this and the authentication is just for that bucket, then is it still considered insecure?
Even having long lived access keys and secret keys anywhere is insecure.
There really isn’t a need for them. If you are running the application on any AWS compute - EC2, Lambda, ECS, EKS, etc, there is an IAM role attached to the VM that gives code permission to run.
On the client facing side, it should be connected to your Orgs SSO solution so when a person leaves the company, you deactivate the user in one place.
Besides, I can’t think of any organization of even 2 people that isn’t already using Office365/OneDrive or Google/GSuite with plenty of shared storage. The cost per seat for either is $6 - $25 per user.
I guess this is for smaller organizations with no MS365 subscription (and thus access to SharePoint)?
Does MS365 cover all potential use cases, needs and scenarios?
All in the world? I suppose I haven't tried it for recipes, but I guess Copilot could help with that too.
As far as sharing files goes though, yes. I mean, it allows you to... share files, and do so in a controlled manner. Even edit them in-app as long as they're of a supported format.
I obviously wouldn’t put my key into a third-party site. However, is there any concern with the self-hosted solution? Looks pretty cool and wouldn’t mind using it. Just make a key specifically for this?
Yes, feel free to try open-source version available on Github.
Just published a new release to support multiple cloud storage options.
Thanks
Hi! I’m the author of another open-source project in the same space that I’ve been working on for a while: Filestash [1]
Were you familiar with Filestash before starting this? If so, was there something specific you felt was missing that inspired you to create your own solution? Would love to hear your thoughts.
github: https://github.com/mickael-kerjean/filestash
demo on s3: https://demo.filestash.app/login?type=s3&access_key_id=Q3AM3...
Your reply comes off as arrogant and hijacking someone's announcement like this is tacky. "Were you aware I had already done this better. Why would you do something I already did? Did I mention I already did this?"
Never came across this, Interesting project. Thanks for sharing
Nice!
Could you tell me why Google Drive didn’t work for you?
Google Drive seems to cheaper and has better UX than S3 - $8 for 2 TB (India) - Supports file versioning - 750GB bandwidth per day
If you do any less typical files Google Drive will balk at you at all times, at least for the other users who use the browser UI.
What works bad: * folder with lots of medium size files * large ZIP * video bigger than a few tens of megabytes
New Release is out now
> We support 7 cloud providers now. > Beta storage providers are still being tested.
Stable: Amazon S3 ($23/TB/month) Google Cloud Storage ($20/TB/month) Cloudflare R2 ($15/TB/month)
Beta/Testing: Wasabi ($6.99/TB/month) Backblaze B2 ($6/TB/month) DigitalOcean Spaces ($5/TB/month) Hetzner Storage ($4/TB/month)
Why should I use your solution and not Next cloud, for example?
Our S3 File Manager offers a lightweight, zero-configuration solution focused purely on S3 storage management, making it ideal for teams who need simple, cost-effective cloud storage without the overhead of a full collaboration suite like Nextcloud. While Nextcloud excels at comprehensive collaboration, our tool excels at simplicity and AWS integration. But we don't want to limit it to S3. We will try to include all storage by next month.
Hey, looks interesting and practical.
Any plans on making it compatible with other s3 implementation (other cloud vendors, local minio.io etc)?
Yes! Will release next version which supports all cloud providers from S3 to Hetzner Cloud storage!!
That is really neat. I had so many tiny use cases in my previous companies that could be solved by just a simple UI like this one. If you combine that with things like s3 static websites, it could be a beast that replaces some long-forgotten CMS solutions.
Completely Agree. I would appreciate it if you could add your ideas as an issue here: https://github.com/rohitg00/s3-file-share-for-free/issues
[dead]