I can't help but imagine this type of audio finesse, and the realism it would add to VR.

Audio engineer here. Lower frequencies tend to propagate more evenly and it's crucial for distance perception. Let's say we have a speaker playing a full range song. Higher frequencies come out in more narrow cone, while lower frequencies come out way more evenly in all directions and that's where the losses are. We use this effect to place instruments in the mix, adding/removing the "presence" effect. It really sells the distance.

Ironically, the whole visualizing sound thing could lend itself to a Dare Devil game someday.

Another thing to take into account is that lower frequencies should penetrate materials more than higher frequencies. Another interesting thing would be to take into account for diffraction, where lower frequencies also diffract more than higher frequencies, but the effect may not be worth the added complexity/compute. Maybe could split each ray after some number of steps or passing through a small opening into multiple rays pointing in a similar direction with a diffraction angle offset.

Everyone's already said how amazing the raytracing is, but I want to say how absolutely GORGEOUS your visuals are. They are so fluid and organic, it feels like I'm watching a futuristic holodeck come into and out of reality every time the floor is created, it's phenomenal and so beautiful I can't even comprehend how long it all took. And the fact you wrote it all yourself? So cool. This above anything else inspires me to continue programming

Audio engineering is so much underrated because it is not flashy

the deaf accessibility is incredible, alongside everything else of course

I made an audio ray tracer a while back which did exactly this, but used more from the tracer to get even more realistic audio. Not only did I get delay, echo and volume change, I got resonances, muffling, reverb etc. for free. Every effect could be represented using this. The algorithm which takes the ray tracing data and produces highly realistic sound, as if it was played in real life, doesn't use any approximation or panning depending on the average direction. I called the algorithm DCDBR (Dual-Channel Dynamic Buffer Redistribution) and all it does, no trickery, is just distribute buffers of audio in a larger buffer depending on the ray tracing data. There is nothing really fancy about it either. The ray tracer also accounts for the materials absorption and I've thought about adding transmission too.

Play around with an effect called “comb filtering”. It fundamentally recreates what we hear when sound reflections collide- some frequencies add together, some cancel each other. With a light touch, it should feel so familiar that it’s like a cheat code. The point of diminishing returns comes quick. You’re onto something great.

This is incredible stuff man, especially helping the deaf "see" sounds. People often forget how important sound is in games, love to see this.

This sounds like what Aureal was trying to do back in the early 2000's. It was so good in fact that that Creative Labs sued Aureal out of business, and then bought up all their assets for next to nothing. Had Aureal not been a startup and had the funds to fight the lawsuit, they would very likely have won, as Creative's premise was that "3D audio" was proprietary to them alone. You don't need a law degree to know that was clearly ridiculous, but it worked.

I love the fact that you considered deaf people and then even colorblindness on top of deafness

To the claim at 1:00: The voice wouldn't sound deeper. The pitch would stay the same, but the level of higher frequencies would be lower, because bass isn't blocked as well as mids and highs through walls and doors.

As an audio engineer, there is one thing that was seemingly overlooked, that I haven't seen anyone else address, and that would be the dropoff of your "echo". The rays seem to live forever, when they should be weakening, due to the fact that soundwaves gradually lose energy, even more so when they reflect off a surface.

1:10 it was very rude of me not to turn qround while Jamie was speaking to me. Sorry, Jamie :(

I'd rather have raytraced audio than raytraced graphics