See the ray_cube_compute, ray_cube_fragment examples which use the wgpu api built on top of the wgpu-hal api.

I implemented it for the most part: https://github.com/Lichtso/wgpu/tree/metal/ray_tracing_acceleration_structures
But it does not work yet. It can be compiled and cargo run --bin wgpu-examples ray_traced_triangle runs, but there are no intersections / hits. Probably missing a encoder.use_resource(blas); somewhere.

A challenge I encontered is that metal does not use raw GPU pointers like all other backends. Instead one has to build the BLASes first and then add a list of their handles to the TLAS descriptor to build it. Then the instances in the TLAS can reference those listed BLASes by index. This could require a rework of the AccelerationStructureInstances across all backends I think.

And then there are these other annoyances like how the flags and the barriers are handled.

This could require a rework of the AccelerationStructureInstances across all backends I think

Maybe try MTLIndirectInstanceAccelerationStructureDescriptor? https://developer.apple.com/documentation/metal/mtlindirectinstanceaccelerationstructuredescriptor.

Edit: Also see https://developer.apple.com/documentation/metal/mtlindirectaccelerationstructureinstancedescriptor.
Edit 2: It seems to (unclear from docs) be possible to continue using MTLAccelerationStructureInstanceDescriptor without setting instancedAccelerationStructures by setting instanceDescriptorType to indirect and using MTLIndirectAccelerationStructureInstanceDescriptor in the buffer.

Great idea, I will give it a shot after I got something working. I fixed a few more things, the metal API validation and the msl shader validation pass, but so far there still are no intersections ever at all :(

At least I can now see the BLAS, the TLAS instantiations thereof and the shaders acceleration structure buffer binding being active in XCode: https://developer.apple.com/documentation/xcode/inspecting-acceleration-structures

Still, the actual shader produces no hits even if I make all parameters the least restrictive possible (like using a negative minimal distance, forcing opacity and full hit mask etc.).

The only other thing I could find is that you bitshift custom_data into the mask (I don't think this would fix it but maybe try?). IIUC rather confusingly, I think that the place custom data should go is in UserID (in MTLAccelerationStructureUserIDInstanceDescriptor, or in the indirect descriptor). This isn't actually the ID of a user but a user provided ID for the instance. This has a confusing name, but is 8 bits sent to the shader and not used anywhere else which is what we need. The only other idea I have is that the rays are going is the wrong direction (e.g. 90 degrees off). Have you tried aligning the acceleration structure viewer camera to the ray direction?

Well, turns out it was already working. I just had Xcode configured not to run cargo build, so I was looking at an outdated build.