Bandwidth…
Freesound is using around 3 to 5 TeraByte per month these days, being capped at 2 MegaByte/sec. This upper limit is imposed by the university. UPF still runs a 100 mbps network as far as I know, which means that with 2MB/sec (=16mbps) we’re using 16% of the whole university bandwidth if we are pushing the upper boundary! That’s pretty much all the time now.
But, where does this bandwidth go to? Today I did an analysis of the last full 7 days to see which of the 4 would take up the most: sound packs, sound images (those colorful waveform displays), sound previews (the mp3 your hear when you hit play) or actual sounds! The total bandwidth used by these categories is 694.37GB, or split up:
- sounds 40.4% (280.63GB)
- previews 36.1% (250.63GB)
- packs 22.4% (155.61GB)
- images 1.1% (7.51GB)
First of all, we can completely ignore the images. They are not our problem. PNG is a very nice format for images with few colors, and our waveform display are just that. Other than that it looks like sounds, previews and packs are… well, more or less using the same amount of bandwidth. I’m a bit surprised that packs are only 22%, I thought people liked packs a lot more than files. Let’s see what happens if we split up the sound category:
- wav 77.1% (216.34GB)
- aif 9.2% (25,96GB)
- mp3 8.2% (22.88GB)
- flac 5.3% (14.98GB)
- ogg 0.2% (0.47GB)
People really like uncompressed sound and we have a lot of it at Freesound: wave files lead the way with a huge 77%!
Flac…
Free Lossless Audio Codec (FLAC) is a file format for audio data compression. Being a lossless compression format, FLAC does not remove information from the audio stream, as lossy compression formats such as MP3, AAC, and Vorbis do. (taken from http://en.wikipedia.org/wiki/Flac)
What would happen if we would compress all wave files to flac (thanks to Nico for bringing this up so I had an excuse to do this analysis)? Flac has an average compression ratio of about 0.6 (1MB wave file > 0.6MB flac file) so we would save about 550GB per month which is quite a bit!
On the other hand flac is a bit of a pain to support: many people don’t know flac and like to be able to use their wave files immediately after downloading. There’s no visual guides for people how to use flac, nor is there a cross-platform unified interface for using flac. A great format, but not very user-friendly if you don’t like the command line! Flac developers, prove me wrong and I will make freesound the biggest proponent of flac!
I’d also like to add that flac is a free format, and unencumbered by patents (unlike mp3, for instance…), which I think fits the Freesound spirit quite well 🙂
Flac also support tags, which wav doesn’t.
The biggest problem I think is native flac support inside music applications. Some of them already support it, such as nearly every Linux app, Ableton Live, and probably some more which I don’t know about, but there sure are some other applications that do not load flac files, which is a shame…
So if your favorite app doesn’t support flac yet, this might be the time to send a mail to their author(s) requesting that they implement flac support!
Go for FLAC, if you’re not going to introduce it to the people, who will? Also there are easy tools (for example foobar2000) or even the command-line-tool. People who are serous about sounds are likely to know it and if not, likely to be wanting to learn it.
Another idea: When people press “play” on wav-files’ previews, are they getting the WAV served or only a reduced mp3 version? the latter would make better sense.
Another another (but stupid) idea: Auto-transform wavs to flac and ogg and allow people to choose to either use ogg vorbis or flac files. There might be enough people who don’t care about the small quality increase of using flac (for example people, who don’t want to edit the sound files).
I would agree with these comments.
Personally speaking I’ve been using FLAC for important file transfers for a while (I’m in Canada and often swap audio data with a colleague in the UK). I’ve found it to be an excellent format. It’s easy to use and, in particular, batch converters are available making the encoding or unencoding of large amounts of files no problem.
I’m going to consider uploading all my future Freesound files in FLAC format.
With kind regards,
Sandyrb.
Yesterday I was thinking about this again, and there’s one big problem though: flac does not support advanced formats like BWF or even wave loop points as far as I know. When you encode a BWF into flac, you loose the additional data. So, while flac is a perfect format for compressing music without losing quality, it could be better as a professional format.
I just found this (flac_options_keep_foreign_metadata) but that doesn’t really fix the problem because at the end-user side of things the user needs to decode with the same command line options, making it -again- difficult.
qubodup: all the previews are 64kbps mp3’s. Pitty flash does not support Ogg! As described in the article, Flac is not “monkey-proof”: WAV is because it’s supported by every program. So, until that “critical mass” of users that use Flac is reached, a monkey-proof interim solution needs to exist.
I’m still not entirely convinced, but I like qubodup’s idea to store in 2 formats (one lossy and one Flac), but then we could have the same discussion about ogg 🙂
I doubt that ogg and flac will become monkey-proof in the next ten years, but they might be about audio-monkey-proof. Isn’t it so that freesound is a site for audio-monkeys? (I wouldn’t know..)
PS: I didn’t even know about metadata being existent/possible in wav files! That surely sounds like serious business. I can’t think of a sane solution. Only of adding “DECODE WITH –keep-foreign-metadata” in the file name or writing a note on the download page that there is metadata to be recovered (if any was detected).
There’s of course the suckey solution of zipping all lossless wav etc. files for download………. 🙁
Hey why don’t you put an option in user preferences “Download all WAV/AIF files as flac”. Convert them on the fly when clicking download. Make it obviously visible in registration and stuff.
I would love it, would make my downloads much faster and would save me some HD space. I use Renoise for working with sounds, and it supports flac out of the box. As an added bonus, you save some bandwidth. 🙂
Suva,
that would be great, but in exchange it would impose a huge CPU load on the server, which would be encoding hundreds files… again and again!
In general I use this rule of thumb: is the file is larger than 8-10 MB I upload as flac, otherwise as wav. The logic behind is that anyone not discouraged by the size of a large sample would not be discouraged about learning to deal with a flac decoder either
My 2 cents
D
Well, you can cache the encoded files in some amount so you don’t need reencoding, but this takes more storage at the same time.
Caching flacs on disk as would mean 70GB more hard drive space needed. Not a bad idea, but I think 9/10 people would prefer to download wav.
Pingback: micablog » freesound’s bandwidth
Maybe you could get someone to do client-side uncompression. That way you could store and transfer the small files and the user would get the WAV. Maybe you could Java for this if Flash does not help. Not sure about the support for saving files from an Applet these days though 🙁
wkr,
gkokmdam
I say go with flac
Suva: On the fly is ridiculous, cpu power is more precious than storage, but allowing the choice might be a good idea.
gkokmdam: Anything but direct download is probably a bad idea, using a java-app for downloading a file sounds like overkill
admin: 9 of 10? You actually could run a test phase, in which people would be allowed to choose between wav and flac (and ogg vorbis?)
A kind of info text could be in proximity of the download links, introducing flac-supporting software or open source flac converters, or telling stuff like “flac is an open format which compresses wav files to a smaller size without loosing data” etc (sorry for the catchphrase “open format” I will wash my fingers in a second)
I developed FLAC so let me address some points:
1. no unified GUI across platforms for FLAC: that’s true for any codec, the closest thing is itunes for osx and windows only. if you want itunes to support flac, join the petition: http://flac.sourceforge.net/itunes.html
2. no native FLAC support in apps: pretty much everything except itunes and wmp support flac. production: ableton live, soundforge, vegas pro, audacity. players: winamp, foobar, play, cog, dbpoweramp, songbird, vlc, xmplay, yahoo music engine, xmms, mplayer, xine, on and on.
many more:
http://flac.sourceforge.net/download.html
http://flac.sourceforge.net/documentation_tasks.html
http://flac.sourceforge.net/links.html#software
3. doesn’t support BWF or wave metadata: it actually does (I see you found the option), it is not on by default in flac.exe because most people don’t want the wasted space. however it is available and on by default in e.g. sounddevices recorders where it counts.
4. you may not know but FLAC is already the lossless distribution format of choice for many labels, records, and artists from metallica to phish. here’s a small subset:
http://flac.sourceforge.net/links.html#music
you may have heard, NIN just released ghosts i-iv in FLAC, not wav. there are entire backend store software systems where FLAC is built-in, used to sell concert recording by many many bands, not no-names, but people you’ve heard of.
There is a GUI program that make by illustrare called dBpoweramp Music Converter.
The converter have support for flac by downloading additional free plugins offered by the company.
Besides, it is free too. But, it is not open source though.
hello jc, thanks for joining us… this is becoming quite a discussion I’m starting to think we should be moving it to the forums 🙂 Anyway, let me reply to your points.
1. true. My original post expresses it wrong: what is really needed is an open source, cross-platform audio converter with a very simple GUI. Something that can load a file and convert it to another file, with no special options (except one button which says “advanced mode” 😉 ). This has nothing to do with flac, ogg or any format, but I guess it would be nice if the flac website DID have a simple, open source, cross platform “flac->wav->flac” utility with GUI (the same goes for OGG or MP3!). However…
2. this is a very valid point and does nudge me a bit closer to using flac for downloads. The only people that won’t like it is the people that don’t really know much about audio, but just want to use a file in powerpoint,, mobile phones, … I guess we could supply them with 160kbps mp3 version of the files, for example. If sizeof(flac + mp3) < sizeof(wav) we're still winning on both hard drive space and download bandwidth.
3. is there any possibility to make this option in decoding "default on" if the output file supports it and the input file has the metadata? This would be the smart option afaic and would silence the real pros on the site if we decided to switch to flac.
4. yes, very true, and very nice, but freesound isn't a label, it's a site with audio snippets of all formats, shapes and sizes. The vast majority of the people coming to freesound don't want the audio files just to listen to them. So this is a very different crowd with different demands. "Hey why can't I load this in powerpoint" is not something Warp/... would have to deal with normally speaking.
At bdejong above:
There actually is a simple GUI based utility on the FLAC site. It works perfectly with the default settings. And it is available for the three most common operating systems used in audio: Windows, Mac and Linux.
Are you guys using mod_deflate or something similar to send the files gzipped? Its browser support is ubiquitous it and WAV files are quite compressible this way.
I’d like to add my 2cents:
why are we dismissing .zip so fast? I have to deal with zipped (or rarred for that matter) files all day long and it is absolutely no problem for. If people are serious enough to know the benefit of uncompressed vs lossy audio, they will know how to work a zip file as well. Plus, zipping could be done on-the-fly, right? Or would this be too much cpu load as well?
At Tim, above:
ZIP does not actually perform better than FLAC when it comes to audio, because FLAC is designed specifically for audio. The speed and compression are both inferior when using ZIP, especially when you consider that FLAC can be decoded on the fly and read just like any other file, whereas ZIP has to be decompressed first. Here’s an article that includes, among other things, a comparison between the two.
I vote for FLAC.
One of the successors to CoolEdit is Goldwave – it can handle FLAC, so can Winamp (playing them on the fly) and Foobar.
Just a help page on the site, with links to these and similar utilities should be sufficient. Most folk are used to “problem solving” these days with their file formats. And an endorsement of FLAC from this site would only make ubiquitous support better in the long run.
FLAC’s a great format.
🙂
My 2p’s worth.
jmmp
do FLAC, even if it isn’t supported by certain music creation software I use http://www.dbpoweramp.com/ and it does the business! Free the bandwidth!!
Paul
Automatically encode each uploaded file to Flac and offer it as an alternative for downloading.
Audacity is cross platform, open source and supports flac.
http://en.wikipedia.org/wiki/Audacity
Zip is better than nothing and is understood by all?
digifish
I would go the way of converting all WAVs to FLAC. After all, Freesound is a free service, so people can put a little effort in converting to WAV again after download. Which can e.g. be done with Audacity without the command line.
However, FLAC can’t contain floating point WAV data. Which is sort of a drawback but also pointing out that WAV is some weird container format for basically everything. Which again means that with FLAC, you’re on the safer side as far as a clearly defined format is concerned.
Btw, Freesound would not be the first site to use FLAC. Have a look at Jamendo.com! There you must upload your music in FLAC format. They offer a tool for automatic conversion, though. Downloads are offered as MP3 and OGG.
We all know FLAC and OGG Vorbis are the open source way to handle things, they are the best choice. So let’s help those nice formats to spread out! (I would jump for joy if Flash Player finally had OGG support…)
I prefer wma more than flac. Flac is much bigger, but of course better quality. But for my ear, good enough is wma.
As another example, I compressed about 11k wav files and the flac files were about 35% of the total size of the wav files.
I think the suggestion to have both wav and flac is the best option, if disk space is not a problem. That way, you maintain the wav for new users and flac for experienced users. Some online archives (like the Internet Archive) offer the same files in several formats, the new users download the formats they know, while more experienced can download the “better” format, in this case, flac.
Also, you can “sell” flac by showing that it takes less time to download.
wav > flac > ogg
the incredible shrinking process.
i made a wav file using audacity in ubuntu linux. it output to 3.6 MB
i opened that wav in audacity and converted it to flac and added tag information. it output to 2.6 MB
i opened that flac file in audacity and converted it to ogg (with the same tag information, it was already there i didn’t need to edit it or add it). it output to 0.4 MB.
i opened that ogg file in audacity and the tag information was still present. thus, from the outside the files look identical and sound identical (i couldn’t tell a difference, but then again i am not a dog).
what then is the difference between that ogg file and the flac file besides the obvious 85% smaller file size?
is the ogg not lossless like the flac?
why would we be pushing flac when ogg is way smaller?
if neither are fully supported everywhere, why not push ogg?
is the ogg file really a flac file, and the ogg container is just way smaller than the flac container?
confused… =)
ogg is lossy not lossless…
– bram
dammit, i just read the ogg documentation and was going to post my findings before someone else exposed my foolishness, but bdejong beat me to it. thanks for your reply, i get it now =)