Nice breakdown. I've been wondering why it's so complicated to program on the GPU compared to the CPU. Is it a language paradigm issue or maybe a hardware design limitation? I always think of this when looking at the graphic APIs, espeically Vulkan.
It's not more complicated, you have things like CUDA which is basically C/C++ that compiles to GPU.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
Right, that makes sense. There's a ton of overhead in the setup though, I'm assuming that's because everything runs on the CPU by default. Is this then an OS design choice? Thanks for your input.
There can be overhead, though GPU memory + drivers are some of the most optimized software in the world.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
For me, if-branching is the hardest to wrap my head around. "If"s are so natural to CPU programming that it's second nature to any developer but on GPU, it's expensive.
instead of iterating over an array and if-branching to one of two paths for each item, conceive of your conditionals as logic operations which test for set membership (i.e. "is this array item a member of the set i care about?") and with this array mask of "do the thing or don't do the thing" flags, you can mutate your data 0 or 1 times.
the key for me is thinking in terms of sets—much of your conditional logic is probably about sorting things into sets anyway.
part of the reason is historical cruft & history path dependency. GPUs didn't start as general computing device for parallelism but a device for the specific purpose of accelerating graphics with very specific fixed functions for that job and only that. with time that fixed pipeline became more complex and more configurable, not fixed/more flexible and hacks were in parallel used to make use of it for tasks outside graphics those hacks were later made into something more formal (CUDA/OpenCL).
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
Good summary, thanks. Sounds like it's all still pretty early, even for graphics but much more for parallel compute, which seems the direction the hardware is going in. I'm sure there is a CUDA equivalent for CPU but I would guess it's all the way back in the 70s or earlier. Hopefully open standards gets some traction.
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (...)
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
Sony tried that with the PS3's Cell processor, just a wise parallel vector processor instead of dedicated hardware.
If you knew how to use it, you could get great results, which is how Naughty Dog's games became so spectacular. But if you just tried to port an existing game, things went downhill quickly, which is why GabeN talked on stage how much he hated the PS3.
Oh and this complexity led to lots of games just switching to one of the existing engines that supported it natively, and started the modern dominance of Unreal Engine. But it's also why engines introduced more parallel architectures, which a few years later became very useful when quad-core processors in PCs became mainstream.
> don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
I bet it's all the same if not worse, and that's why GPU drivers are closing in on the gigabyte "sound barrier" (and I guess on Linux that part is played by Proton... e.g. I bet there's a ton of game-specific 'workarounds' in Proton, if not for performance, then at least to fix 3D API misuse).
It's partially historical; I'm sure if we redesigned GPGPU today from the ground up, we'd have an entirely different pipeline. And partially due to hardware vendor lock-in; you can't really design an alternative to Vulkan/d3d12 even if you had the skill and drive to do so.
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
actually yes you could design an alternative API and even implement it....
there's HW references + the linux drivers for AMD (both UMD and KMD), there's refs for NV and their open linux KMD (no UMD though), intel also has HW refs and open drivers for linux
I think it would be possible to put together as long as you don't mind only working for certain hardware, like a prototype you know
I might give it a shot to make a proof-of-concept "high level low level" graphics API ngl if no one else feels like so...
I think there's space for a "two-level" design like have the driver consist of the HW-specific parts, expose things in terms of GPU VAs, explicit sync, explicit memory management, command buffers etc. then have a high-level wrapper over it to provide vaguely GL-ish semantics, and you could intermix the two APIs, obviously you wouldn't be able to get the VA for a high-level object because that'd break the autotracking, if you use low-level you're responsible for syncing, cycling etc.
It's mostly hardware design limitations. GPU hardware is way more diverse than CPU hardware, so you need to distribute shaders as either source code or a portable intermediate representation, in both cases compiled at runtime for the user's actual GPU. There are also many fixed function parts of the graphics pipeline with wildly different behavior on different hardware which you need to abstract over. And it all runs in parallel with the CPU, often with a separate pool of RAM, so you need implicit or explicit synchronization and data transfer which adds a lot of complexity especially in error cases.
If you really want to see why all the cruft is there, and how much you could remove if you dropped support for older hardware, look no further than Sebastian Aaltonen's "No Graphics API" blog post (which he actually implemented too): https://www.sebastianaaltonen.com/blog/no-graphics-api/
The thing that makes it "hard" is that it's all indirect. You compile a program for the GPU (a shader). Then you have to setup data (textures and buffers) for it. Then, you need to spend a bunch of time telling the GPU how to use that data, offsets where each piece of data is, what formats the data is in,, how to advance to the next piece of data, etc.. etc..
There are a few libraries that try to abstract over all this indirection but all the graphics APIs themselves are lower-level so you're left to do all of that yourself.
I think most of it comes down to 'historical accidents' tbh.
E.g. if CPUs would have closed ISAs and heavily guarded IPs, you'd also need a 'driver' and an 'OpenCPU API' to make any use of them (of course the public ISA of a CPU is essentially the driver interface baked into the hardware).
The other big difference is focus on decent single-thread performance for CPUs versus performing thousands of small tasks in parallel on GPUs, but each one slowly. This difference causes a different programming model (e.g. running an entire application in a single 'main' function from start to finish, versus spawning small specialized tasks which are running across hundreds of threads).
And the specific problem with Vulkan is that it tried to solve two conflicting requirements: on one hand being an explicit low-level API without 'magic abstractions', on the other hand supporting very different GPU architectures (especially the mobile GPU zoo). The resulting mess was quite predictable to be honest...
First, GPUs aren't simply "highly parallel CPUs", they also contain a whole lot of fixed function hardware that is specialized for real-time computer graphics. This hardware isn't exposed as "fancy instructions", it is set up in other ways, which is achieved through a device driver.
Second: GPUs are coprocessor, separate from the CPU where another OS is running. This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS. Because, in case you were not aware, the GPU itself runs its own little operating system, although it is rarely called that.
> they also contain a whole lot of fixed function hardware that is specialized for real-time computer graphics. This hardware isn't exposed as "fancy instructions", it is set up in other ways, which is achieved through a device driver.
Those hardwired fixed-function parts are getting fewer and less important though (ok, and then raytracing units entered the scene...). Also AFAIK something like a texture sampling operation is indeed a 'fancy instruction'(?), even though the actual texture sampling is done by a 'fixed function' hardware unit.
> This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS.
I bet this type of GPU/CPU handshaking work could also be baked into the hardware (similar to CPU mirocode) and exposed via a standardized memory mapped IO interface (or whatever the current mechanism is for letting the CPU control the GPU). E.g. a CPU also implements similar protocols to talk to the rest of the computer system (e.g. memory and other CPUs), but this is mainly implemented as "hardware protocols" - drivers "just" offer more flexibility for hardware vendors because they can delegate the standardization into software.
...also a lot of the "CPU / GPU software ecosystem" was probably shaped by CPU and GPU vendors historically being different companies, and once the big consolidation started, the conventions were already set in stone.
> Those hardwired fixed-function parts are getting fewer and less important though
There is more machinery internally than what the user can see. Even "low level" APIs hide quite a lot of the internal complexity.
> Also AFAIK something like a texture sampling operation is indeed a 'fancy instruction'(?)
Yes, and no. The "I want to sample this texture" part looks like a fancy instruction to the shader processor. But there is a fair amount of setup that needs to be done before and after, and that is the responsibility of the device driver. Stuff like: where is the data, how is it arranged in memory, how it is to be sampled, etc.
As you point out, the texture units are coprocessors inside the coprocessor that is the GPU itself. It is turtles all the way down. Think of a GPU as a whole network of processors inside a chip, some of which are hidden from the application programmer.
> I bet this type of GPU/CPU handshaking work could also be baked into the hardware (similar to CPU mirocode) and exposed via a standardized memory mapped IO interface (or whatever the current mechanism is for letting the CPU control the GPU)
Attempts have been made. Last time I checked, they had not been very successful, but I have not kept up with the latest developments after retirement.
The way this normally works is that there is a circular command buffer where the CPU inserts commands and the GPU consumes them. It works; there is nothing wrong with it. It is the obvious way to handle asynchronous coprocessors.
> E.g. a CPU also implements similar protocols to talk to the rest of the computer system (e.g. memory and other CPUs), but this is mainly implemented as "hardware protocols" - drivers "just" offer more flexibility for hardware vendors because they can delegate the standardization into software.
I suspect you underestimate the amount of setup that is necessary to drive a GPU. And the amount of asynchronous resource management involved.
Let me put it this other way: the people doing this for a living for decades know what they are doing. There have been attempts to try to make GPUs that behave more like CPUs in terms of how they are driven, and they have not been very successful so far, to my knowledge. AMD acquired ATI back in 2006, so it is not like there hasn't been enough time for the concept to cook in the oven. See also Intel's Larrabee.
One thing that helped me understand it was to reframe it as a network application, where you have a client running on the CPU and a server running on the GPU (the shaders).
A bunch of the complexity comes from having to write both client and server, and the communication protocol between them, then some more comes from the client being responsible for spinning up the server. As with every network service, client/server traffic is the slowest part of the architecture, so a bunch of complexity then comes from having to minimise the amount of network calls (or draw calls, in this case).
Then you hit the part where all of this is performance critical (soft-realtime requirements), so the whole thing is geared towards “writing your own userspace IP stack in C++” rather than “writing a flask application in Python”.
Vulkan, OpenGL and DirectX make GPU programming more difficult than it has to be for multiple reasons, some of which other people here have mentioned. They are also low level, focused on high performance, and are primarily designed and written for game engines. Making them easy to learn for newcomers is pretty low on the priority list. ShaderToy makes it easy by taking care of everything except the GLSL, you might like it. You can also try CUDA, which is generally a lot easier than Vulkan. For a really easy/fun time with CUDA, try Warp - it’s CUDA in Python.
I've learned more graphics writing software rasterizers and ray tracers than using any APIs. If the goal is to teach GPU rendering specifically, then OpenGL is indeed more amenable to teaching than the others. But if the goal is to teach graphics and rendering concepts, then OpenGL et al are just miserable and counter-productive.
Disagree.
Trying to express all of graphics and rendering through the lens of OpenGL will of course fail and doing 3D rendering itself via raytracing is simply not possible in rasterization focused OpenGL without dirty workarounds. That is obvious.
A fundamental shift in thinking that got me into high performance graphics programming and parallelized image processing, the "Aha" Moment, is being forced to express how to calculate the final pixel independently, without having access to neighboring pixels of the same output buffer, confined to the straight jacket of the fragment shader. OpenGL with it's combination of hand-holding and relative simplicity as compared to Vulkan and limits of a predefined shader pipeline as compared to a pure software implementation is exactly the thing that I needed to get into graphics programming. It's an excellent jump in point I think.
Agree. But compared to the new APIs, OpenGL is better in that regard. I would argue that legacy OpenGL is even better.
You can implement in software whatever you want and just handwave specific parts to GL, like rasterization. The rest of the boilerplate code will be minimal.
In new APIs you are bound to write 100s of lines of boilerplate code to get to the point. In the time you should be learning about barycentric coordinates, you are still learning about descriptor sets. And still, you don't learn about rasterization, texture mapping, clipping, face culling, because they are still fixed pipeline.
The only problem with learning with 100% cpu rendering is that you need some patience before seeing beautiful stuff because you need to learn/implement everything up until that point.
GLES isn't a bad way to teach graphics but I wish there was a middle ground between it and VK.
The worst part of GL is the mysterious global state you will never understand without studying the spec. The world could use an API that's basically GL but without global hidden singleton (per thread!) BS behind the scenes.
dx11 isn't bad but it has a looot of cruft around it, writing a good constant buffer manager is witchcraft (remember, there's no "loose uniforms" injected into the command stream, it's all constant buffer updates, drivers do all sorts of writecombining magic to mitigate it, but the naive map / unmap cycle without any driver optimisations would absolutely tank....
btw you can run dx11 on other platforms because dxvk exists, but while d3d9vk is a solid idea (high-level API, lowering to vulkan often makes it much faster than the crummy MS d3d9 runtime + DDI combo), d3d11vk is just often a wash or even a regression. I've had to debug dxvk-only perf problems a bit too many times...
anyway, point is, it's not really an easy-to-use or intuitive API
> writing a good constant buffer manager is witchcraft
Tbh, I think the main lesson is simply to not try to work around the specific 3D API conventions too much, e.g. don't try to shoehorn mechanisms into the D3D11 layer that would be straightforward in Vulkan or D3D12 but would disable the special fast-paths in D3D11.
E.g. for a traditional "one uniform update per draw call" loop with one small constant buffer per shader updated via plain old UpdateSubresource(), D3D11 is incredibly fast (you can easily do hundreds-of-thousands update+drawcall pairs like this per 60Hz frame on a somewhat recent - e.g. post-2016 or so - mid-level gaming PC.
Of course trying to emulate D3D11 on Vulkan or D3D12 might have all sorts of issues that don't exist on a good "native" D3D11 driver.
Also somewhat related: in a GL shim it's typically still faster to update uniform data via the old glUniform*v calls instead of using uniform buffers, apparently most GL drivers lack the special fast-path optimizations of small D3D11 constant buffers, and the more abstract glUniform calls seems to give the driver more 'wiggle room' to implement an optimal upload path.
I mean sure you can do a generic layer and common denominator stuff, basically the kinda stuff webgpu or those rendering abstractions do and that's fair enough, it works well for simpler stuff!
The problem really is that if you do that you'll end up with random weird stalls or perf problems you can't really debug sadly :\
Re GL ye glUniform is usually much faster because you just do a lot less, you're writing a small amount of data directly into constant memory, it doesn't need resource tracking. UBOs / constant buffers come back to the theoretically slow but can-be-made-faster-with-driver-heuristics approach...i.e. if you see a small UBO which is only used as a UBO, you get the same direct constant bank upload
On NV it's quite fast, on AMD it's not really optimised, on intel just use buffersubdata because everything else sucks....
Nothing is an easy to use intuitive api with all the features and all the performance, it's (Dx11) the sweet spot of where I felt like it was easy to learn and easy to use without worrying about performance too much.
Sure if you are building the newer rendering pipelines DirectX 11 leaves a lot on the table but it brought a lot of nice things with it, Debug Runtime alone was worth moving to it, 99/100 times your app written in DirectX 11 even if not optimised properly will perform better than DirectX 9 in my experience.
This holds with OpenGL though the modern OpenGL 4.6 and stuff wasn't bad, but it's a lot of cruft still.
Either way there are no perfect APIs, a truly general purpose API that just gives you basics like Texture handling and io and data mapping would be enough today imho. Fully general purpose compute APIs are the solution with some additional tweaks to make graphics stuff convenient. IMHO
Though it's definitely much harder than plain and simple paint colors and textures on vertices & geometry uv approach of old.
WebGPU backend (has a couple more demos which use features not available on GLES3.0/WebGL2, like compute shaders): https://floooh.github.io/sokol-webgpu/
I agree with Adam Sawicki that it's not that relevant any more but also I do believe it's valuable lesson to tech students where the modernity comes from.
I just don't see personally sense to focus on it longer than just 1mo of intro, tutorial, get through simple example, show limitations, carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles.
> carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles
Except that's a minefield all of its own. DirectX 12 is its own special beast that is unlike the others. Vulkan is already mired in 3 revisions of architecture changes. WGPU is based on the (outdated) first cut of Vulkan, not the current model. Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
Tbf, it all depends on how deep you look into the fractal ;) The main differences between the modern APIs are in how resource binding is done, but many other concept overlap quite well between Vulkan, D3D12 and Metal (especially in more recent Vulkan revisions).
> Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
...unless you start with Metal 4, which is bringing in a lot of Vulkan/D3D12 concepts (for better or worse, it's definitely not great for 'convenience' but may help with porting Vulkan or D3D12 code over).
Playstation and Nintendo are doing just fine in regards to mindshare, XBox less so, but it has nothing to do with DirectX NDA for it, rather mismanagement.
I agree, this is also where I'm investing my time Metal mostly and then try to adapt it to DX12 on Windows and I totally skipped Vulkan for that simple reason you have described. but in generic terms only Vulkan comes with portability as analogy to original OGL vision.. I respect that even though the whole framework is to be ditched and rethought into something even more modern and sustainable...
so Metal yes, if somebody wants to be Apple focused. but in the academic world I don't believe that's the case either...
Note that in regards to OpenGL original vision, game consoles never supported OpenGL, although some models kind of did support something, so game studios always cared about pluggable backends than targeting the one true API.
Only the FOSS community is so hard focused on Khronos APIs.
TL;DR because WebGL 2 is basically equivalent to OpenGL ES 3.0, making it the most portable GPU API in the world by far, running on everything from laptops and phones to smart fridges and car head units.
Web browsers have also done a lot of behind-the-scenes work on compatibility. The behavior of WebGL 2 is much, much more consistent across platforms than native OpenGL [ES] ever was.
Personally I wouldn't teach OpenGL ES 3.0 or any OpenGL because its state machine (https://webgl2fundamentals.org/webgl/lessons/resources/webgl...) and horrible error system means tracking down bugs is hard. Instead I'd teach WebGPU as it's portable and has great error messaging.
I'm sure people will objected but I'd argue WebGPU is easier to understand, has better errors, has no state machine, works C++, Rust, JS/TS and other languages, and closer to modern GPU programming if you want to move to Metal or D3D12 or Vulkan later.
Further, OpenGL (and ES) are a nightmare of bugs and incompatibilities students might run into that WebGPU fixes.
That said, the article mentions Apple platforms where you can use ANGLE if you happened to have students stuck there.
I mostly agree regarding GL's programming model, but tbf some problems have been fixed by WebGL2 (much better validation and error reporting - e.g. very helpful error messages right on the browser console - and it's probably the least buggy and most portable 3D API implementation of them all, including the current state of WebGPU.
For teaching I would generally use the web browser via WebGL2 or WebGPU, just because those APIs require much less boilerplate in the browser environment to get to your first triangle and the edit-test-loop is pretty much instant.
In real life there’s lots of bugs around and difficult to read error messages.
Re. the boilerplate just have students clone a pre-fabbed repo with the boilerplate handled for them.
The real question should be whether you want students to know how to program in a low programming language or not. Whether they should see what the reality of programming is or not.
If they can’t program in C/C++ and debug the error messages how are they going to survive in the wild? I guess they’ll use an LLM.
> If they can’t program in C/C++ and debug the error messages how are they going to survive in the wild?
The problem is that you don't get *any* error messages from native GL in the first place (unless you wire up a debug-message callback via the optional KHR_debug extension which isn't available everywhere), and the builtin error system via glGetError() is pretty much completely useless.
Other APIs don't suffer from this problem, D3D, Vulkan and Metal all have excellent validation layers which give very helpful messages, and WebGL or WebGPU running in browsers basically implement such a validation layer out of the box.
small nit, that's not true, since GL4.3 the modern debug message callback is core so you can rely on it. Sure that won't save you on MacOS but if you can target modern OpenGL this isn't a problem in practice
WebGPU will however always suffer from being an API designed for the Web, and targeting 2015 thereabouts hardware.
Better than OpenGL, but don't expect that much improvements coming into their way, other than what the Chrome team might feel like pushing, as I doubt Firefox or Safari teams have much left to say about where it goes beyond version 1.0.
The rigid resource binding model via BindGroup objects in WebGPU is already outdated (and funny enough, an area where WebGPU is actually slower than WebGL2, at least when assuming that bindings need to change frequently and unpredictably).
This was a restriction of the Vulkan 1.0 API and modelling the WebGPU resource binding model around that was a quite predictable problem. It would have been better to come up with a slim abstraction that might cost some performance on Vulkan (which is only really relevant on Android phones and Linux desktops), but at least would not slow down WebGPU across all platforms.
And now even the Vulkan maintainers have realized the problem and are introducing the much more flexible descriptor-heap extension (about a decade late, but better late than never).
WebGPU is essentially the frozen cross-section of 3D APIs from about a decade ago, and the state of GPU hardware from about 15 to 20 years ago (but tbf, apart from the broken binding model there wasn't much that could be done, because most restrictions are coming from mobile GPUs which a browser 3D API simply must support).
1. monolithic pipelines, pretty much no hardware works like that, the hash-and-cache design makes your design inflexible and adds a ton of overhead
2. WGSL, like come on
3. Hilariously outdated feature set, no mesh shaders, no HWRT, cooperative matrices/tensors, etc.
4. It's security-focused because web untrusted code:tm: so most of the calls are hilariously slow. Yes even the desktop implementations. There's no good webGPU implementation out there
5. it's also not simple to either implement or use. it's basically vulkan's difficulty without any of the advantages....
The tutorials and hands on coding examples made me understand how to program with GPUs. I loved them because at the time, khronos was still in its infancy and just started out with GLES extensions and the whole WebGL ecosystem.
Meanwhile I kind of believe that learning where it came from (fixed function pipeline, buffer/texture and uniform problems etc) makes a lot of sense to learn first so you start thinking in shader computation pipelines much easier, as you understand the whys and whens much better.
I probably owe my career to him; it was learning the old fixed pipeline OpenGL API from his tutorials that led to me making games, and those games that led to a family member encouraging me to interview for an internship at his company. All that led to a 20 year career (so far) in software engineering.
I have found OpenGL 4.5/4.6 to be a sweet spot for students currently learning graphics in a semester course. SDL_GPU is otherwise a good intermediate I would consider.
I remember for myself: learning a new API (opengl), new paradigm (gpu programming), and new language(glsl) all on top of learning graphics concepts is a lot. OpenGL 4.5 and 4.6 are modern enough to get started and there are good resources to learn the 3.3+ stuff that I have built on.
See my website(mshah.io) for my paper on some of the 'why'.
Original poster here! Funny enough, I am preparing my new module to be taught in modern Vulkan (with a twist). Looking forward to share the results about this transition.
It is vexing to see ports of N64 games and Quake requiring OpenGL 4 (or better) hardware for no discernible reason other than the version number is bigger. Eyes on you, Ironwail.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
That's >50% of the problem.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
the key for me is thinking in terms of sets—much of your conditional logic is probably about sorting things into sets anyway.
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
Sony tried that with the PS3's Cell processor, just a wise parallel vector processor instead of dedicated hardware.
If you knew how to use it, you could get great results, which is how Naughty Dog's games became so spectacular. But if you just tried to port an existing game, things went downhill quickly, which is why GabeN talked on stage how much he hated the PS3.
Oh and this complexity led to lots of games just switching to one of the existing engines that supported it natively, and started the modern dominance of Unreal Engine. But it's also why engines introduced more parallel architectures, which a few years later became very useful when quad-core processors in PCs became mainstream.
I bet it's all the same if not worse, and that's why GPU drivers are closing in on the gigabyte "sound barrier" (and I guess on Linux that part is played by Proton... e.g. I bet there's a ton of game-specific 'workarounds' in Proton, if not for performance, then at least to fix 3D API misuse).
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
there's HW references + the linux drivers for AMD (both UMD and KMD), there's refs for NV and their open linux KMD (no UMD though), intel also has HW refs and open drivers for linux
I think it would be possible to put together as long as you don't mind only working for certain hardware, like a prototype you know
I might give it a shot to make a proof-of-concept "high level low level" graphics API ngl if no one else feels like so...
I think there's space for a "two-level" design like have the driver consist of the HW-specific parts, expose things in terms of GPU VAs, explicit sync, explicit memory management, command buffers etc. then have a high-level wrapper over it to provide vaguely GL-ish semantics, and you could intermix the two APIs, obviously you wouldn't be able to get the VA for a high-level object because that'd break the autotracking, if you use low-level you're responsible for syncing, cycling etc.
If you really want to see why all the cruft is there, and how much you could remove if you dropped support for older hardware, look no further than Sebastian Aaltonen's "No Graphics API" blog post (which he actually implemented too): https://www.sebastianaaltonen.com/blog/no-graphics-api/
There are a few libraries that try to abstract over all this indirection but all the graphics APIs themselves are lower-level so you're left to do all of that yourself.
https://docs.swmansion.com/TypeGPU/
https://github.com/google/toucan
E.g. if CPUs would have closed ISAs and heavily guarded IPs, you'd also need a 'driver' and an 'OpenCPU API' to make any use of them (of course the public ISA of a CPU is essentially the driver interface baked into the hardware).
The other big difference is focus on decent single-thread performance for CPUs versus performing thousands of small tasks in parallel on GPUs, but each one slowly. This difference causes a different programming model (e.g. running an entire application in a single 'main' function from start to finish, versus spawning small specialized tasks which are running across hundreds of threads).
And the specific problem with Vulkan is that it tried to solve two conflicting requirements: on one hand being an explicit low-level API without 'magic abstractions', on the other hand supporting very different GPU architectures (especially the mobile GPU zoo). The resulting mess was quite predictable to be honest...
First, GPUs aren't simply "highly parallel CPUs", they also contain a whole lot of fixed function hardware that is specialized for real-time computer graphics. This hardware isn't exposed as "fancy instructions", it is set up in other ways, which is achieved through a device driver.
Second: GPUs are coprocessor, separate from the CPU where another OS is running. This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS. Because, in case you were not aware, the GPU itself runs its own little operating system, although it is rarely called that.
Those hardwired fixed-function parts are getting fewer and less important though (ok, and then raytracing units entered the scene...). Also AFAIK something like a texture sampling operation is indeed a 'fancy instruction'(?), even though the actual texture sampling is done by a 'fixed function' hardware unit.
> This also necessitates a device driver to manage scheduling, synchronization, etc. with the main OS.
I bet this type of GPU/CPU handshaking work could also be baked into the hardware (similar to CPU mirocode) and exposed via a standardized memory mapped IO interface (or whatever the current mechanism is for letting the CPU control the GPU). E.g. a CPU also implements similar protocols to talk to the rest of the computer system (e.g. memory and other CPUs), but this is mainly implemented as "hardware protocols" - drivers "just" offer more flexibility for hardware vendors because they can delegate the standardization into software.
...also a lot of the "CPU / GPU software ecosystem" was probably shaped by CPU and GPU vendors historically being different companies, and once the big consolidation started, the conventions were already set in stone.
There is more machinery internally than what the user can see. Even "low level" APIs hide quite a lot of the internal complexity.
> Also AFAIK something like a texture sampling operation is indeed a 'fancy instruction'(?)
Yes, and no. The "I want to sample this texture" part looks like a fancy instruction to the shader processor. But there is a fair amount of setup that needs to be done before and after, and that is the responsibility of the device driver. Stuff like: where is the data, how is it arranged in memory, how it is to be sampled, etc.
As you point out, the texture units are coprocessors inside the coprocessor that is the GPU itself. It is turtles all the way down. Think of a GPU as a whole network of processors inside a chip, some of which are hidden from the application programmer.
> I bet this type of GPU/CPU handshaking work could also be baked into the hardware (similar to CPU mirocode) and exposed via a standardized memory mapped IO interface (or whatever the current mechanism is for letting the CPU control the GPU)
Attempts have been made. Last time I checked, they had not been very successful, but I have not kept up with the latest developments after retirement.
The way this normally works is that there is a circular command buffer where the CPU inserts commands and the GPU consumes them. It works; there is nothing wrong with it. It is the obvious way to handle asynchronous coprocessors.
> E.g. a CPU also implements similar protocols to talk to the rest of the computer system (e.g. memory and other CPUs), but this is mainly implemented as "hardware protocols" - drivers "just" offer more flexibility for hardware vendors because they can delegate the standardization into software.
I suspect you underestimate the amount of setup that is necessary to drive a GPU. And the amount of asynchronous resource management involved.
Let me put it this other way: the people doing this for a living for decades know what they are doing. There have been attempts to try to make GPUs that behave more like CPUs in terms of how they are driven, and they have not been very successful so far, to my knowledge. AMD acquired ATI back in 2006, so it is not like there hasn't been enough time for the concept to cook in the oven. See also Intel's Larrabee.
A bunch of the complexity comes from having to write both client and server, and the communication protocol between them, then some more comes from the client being responsible for spinning up the server. As with every network service, client/server traffic is the slowest part of the architecture, so a bunch of complexity then comes from having to minimise the amount of network calls (or draw calls, in this case).
Then you hit the part where all of this is performance critical (soft-realtime requirements), so the whole thing is geared towards “writing your own userspace IP stack in C++” rather than “writing a flask application in Python”.
Generally GPUs punish you more by bad performance if you don't optimise your code.
A fundamental shift in thinking that got me into high performance graphics programming and parallelized image processing, the "Aha" Moment, is being forced to express how to calculate the final pixel independently, without having access to neighboring pixels of the same output buffer, confined to the straight jacket of the fragment shader. OpenGL with it's combination of hand-holding and relative simplicity as compared to Vulkan and limits of a predefined shader pipeline as compared to a pure software implementation is exactly the thing that I needed to get into graphics programming. It's an excellent jump in point I think.
You can implement in software whatever you want and just handwave specific parts to GL, like rasterization. The rest of the boilerplate code will be minimal.
In new APIs you are bound to write 100s of lines of boilerplate code to get to the point. In the time you should be learning about barycentric coordinates, you are still learning about descriptor sets. And still, you don't learn about rasterization, texture mapping, clipping, face culling, because they are still fixed pipeline.
The only problem with learning with 100% cpu rendering is that you need some patience before seeing beautiful stuff because you need to learn/implement everything up until that point.
The worst part of GL is the mysterious global state you will never understand without studying the spec. The world could use an API that's basically GL but without global hidden singleton (per thread!) BS behind the scenes.
Look at a GDC talk (https://d29g4g2dyqv443.cloudfront.net/sites/default/files/ak...), efficient dx11 is also "reinvent a diy memory allocator" level except that you don't have much control.
btw you can run dx11 on other platforms because dxvk exists, but while d3d9vk is a solid idea (high-level API, lowering to vulkan often makes it much faster than the crummy MS d3d9 runtime + DDI combo), d3d11vk is just often a wash or even a regression. I've had to debug dxvk-only perf problems a bit too many times...
anyway, point is, it's not really an easy-to-use or intuitive API
Tbh, I think the main lesson is simply to not try to work around the specific 3D API conventions too much, e.g. don't try to shoehorn mechanisms into the D3D11 layer that would be straightforward in Vulkan or D3D12 but would disable the special fast-paths in D3D11.
E.g. for a traditional "one uniform update per draw call" loop with one small constant buffer per shader updated via plain old UpdateSubresource(), D3D11 is incredibly fast (you can easily do hundreds-of-thousands update+drawcall pairs like this per 60Hz frame on a somewhat recent - e.g. post-2016 or so - mid-level gaming PC.
Of course trying to emulate D3D11 on Vulkan or D3D12 might have all sorts of issues that don't exist on a good "native" D3D11 driver.
Also somewhat related: in a GL shim it's typically still faster to update uniform data via the old glUniform*v calls instead of using uniform buffers, apparently most GL drivers lack the special fast-path optimizations of small D3D11 constant buffers, and the more abstract glUniform calls seems to give the driver more 'wiggle room' to implement an optimal upload path.
The problem really is that if you do that you'll end up with random weird stalls or perf problems you can't really debug sadly :\
Re GL ye glUniform is usually much faster because you just do a lot less, you're writing a small amount of data directly into constant memory, it doesn't need resource tracking. UBOs / constant buffers come back to the theoretically slow but can-be-made-faster-with-driver-heuristics approach...i.e. if you see a small UBO which is only used as a UBO, you get the same direct constant bank upload
On NV it's quite fast, on AMD it's not really optimised, on intel just use buffersubdata because everything else sucks....
Sure if you are building the newer rendering pipelines DirectX 11 leaves a lot on the table but it brought a lot of nice things with it, Debug Runtime alone was worth moving to it, 99/100 times your app written in DirectX 11 even if not optimised properly will perform better than DirectX 9 in my experience.
This holds with OpenGL though the modern OpenGL 4.6 and stuff wasn't bad, but it's a lot of cruft still.
Either way there are no perfect APIs, a truly general purpose API that just gives you basics like Texture handling and io and data mapping would be enough today imho. Fully general purpose compute APIs are the solution with some additional tweaks to make graphics stuff convenient. IMHO
Though it's definitely much harder than plain and simple paint colors and textures on vertices & geometry uv approach of old.
There's WebGPU (probably the best option for teaching), D3D11 and Metal.
And (shameless plug): sokol-gfx tries to find a balance between simplicity and modern 3D APIs:
https://github.com/floooh/sokol
Demos with links to C source and shader code:
WebGL2 backend: https://floooh.github.io/sokol-html5/
WebGPU backend (has a couple more demos which use features not available on GLES3.0/WebGL2, like compute shaders): https://floooh.github.io/sokol-webgpu/
Awful.
I just don't see personally sense to focus on it longer than just 1mo of intro, tutorial, get through simple example, show limitations, carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles.
Except that's a minefield all of its own. DirectX 12 is its own special beast that is unlike the others. Vulkan is already mired in 3 revisions of architecture changes. WGPU is based on the (outdated) first cut of Vulkan, not the current model. Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
> Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
...unless you start with Metal 4, which is bringing in a lot of Vulkan/D3D12 concepts (for better or worse, it's definitely not great for 'convenience' but may help with porting Vulkan or D3D12 code over).
Ah, and DirectX on XBox is not the same as on Windows.
so Metal yes, if somebody wants to be Apple focused. but in the academic world I don't believe that's the case either...
Only the FOSS community is so hard focused on Khronos APIs.
Web browsers have also done a lot of behind-the-scenes work on compatibility. The behavior of WebGL 2 is much, much more consistent across platforms than native OpenGL [ES] ever was.
I'm sure people will objected but I'd argue WebGPU is easier to understand, has better errors, has no state machine, works C++, Rust, JS/TS and other languages, and closer to modern GPU programming if you want to move to Metal or D3D12 or Vulkan later.
Further, OpenGL (and ES) are a nightmare of bugs and incompatibilities students might run into that WebGPU fixes.
That said, the article mentions Apple platforms where you can use ANGLE if you happened to have students stuck there.
For teaching I would generally use the web browser via WebGL2 or WebGPU, just because those APIs require much less boilerplate in the browser environment to get to your first triangle and the edit-test-loop is pretty much instant.
Re. the boilerplate just have students clone a pre-fabbed repo with the boilerplate handled for them.
The real question should be whether you want students to know how to program in a low programming language or not. Whether they should see what the reality of programming is or not.
If they can’t program in C/C++ and debug the error messages how are they going to survive in the wild? I guess they’ll use an LLM.
The problem is that you don't get *any* error messages from native GL in the first place (unless you wire up a debug-message callback via the optional KHR_debug extension which isn't available everywhere), and the builtin error system via glGetError() is pretty much completely useless.
Other APIs don't suffer from this problem, D3D, Vulkan and Metal all have excellent validation layers which give very helpful messages, and WebGL or WebGPU running in browsers basically implement such a validation layer out of the box.
The beauty of OpenGL versus say Direct3D is that OpenGL is extensible and thus you can add that feature should you so choose.
Better than OpenGL, but don't expect that much improvements coming into their way, other than what the Chrome team might feel like pushing, as I doubt Firefox or Safari teams have much left to say about where it goes beyond version 1.0.
What improvements are needed? It sees pretty much complete to me and I think it's a good thing for it to be in maintenance mode (ie stable).
This was a restriction of the Vulkan 1.0 API and modelling the WebGPU resource binding model around that was a quite predictable problem. It would have been better to come up with a slim abstraction that might cost some performance on Vulkan (which is only really relevant on Android phones and Linux desktops), but at least would not slow down WebGPU across all platforms.
And now even the Vulkan maintainers have realized the problem and are introducing the much more flexible descriptor-heap extension (about a decade late, but better late than never).
WebGPU is essentially the frozen cross-section of 3D APIs from about a decade ago, and the state of GPU hardware from about 15 to 20 years ago (but tbf, apart from the broken binding model there wasn't much that could be done, because most restrictions are coming from mobile GPUs which a browser 3D API simply must support).
2. WGSL, like come on
3. Hilariously outdated feature set, no mesh shaders, no HWRT, cooperative matrices/tensors, etc.
4. It's security-focused because web untrusted code:tm: so most of the calls are hilariously slow. Yes even the desktop implementations. There's no good webGPU implementation out there
5. it's also not simple to either implement or use. it's basically vulkan's difficulty without any of the advantages....
I, and I suspect many others, benefited immensely from him lessons.
Good times.
The tutorials and hands on coding examples made me understand how to program with GPUs. I loved them because at the time, khronos was still in its infancy and just started out with GLES extensions and the whole WebGL ecosystem.
Meanwhile I kind of believe that learning where it came from (fixed function pipeline, buffer/texture and uniform problems etc) makes a lot of sense to learn first so you start thinking in shader computation pipelines much easier, as you understand the whys and whens much better.
https://nehe.gamedev.net/
Talk about nostalgia!
(what I don't get is why this page is breaking pgUp/pgDown)
I remember for myself: learning a new API (opengl), new paradigm (gpu programming), and new language(glsl) all on top of learning graphics concepts is a lot. OpenGL 4.5 and 4.6 are modern enough to get started and there are good resources to learn the 3.3+ stuff that I have built on.
See my website(mshah.io) for my paper on some of the 'why'.
It is vexing to see ports of N64 games and Quake requiring OpenGL 4 (or better) hardware for no discernible reason other than the version number is bigger. Eyes on you, Ironwail.