The neutrons really are the main concern.
The water is important. It turns out that particles transfer their momentum most effectively to other particles of similar mass. Hence the hydrogen in the water molecules is more efficient at slowing down neutrons than a similar mass of concrete. Or even lead. And once the neutrons are slowed down (Thermalized) boron atoms have a very large capture cross section.
The point being that the x-ray conversion system may not be that good at slowing down neutrons as it will be made of heavier (Higher z) materials.