Problem is, to make fusion this way you would have to somehow efficiently convert the gamma ray energy into thermal energy. Otherwise, most of the highly penetrating gamma rays would just zip right out of the nuclear fuel and travel a zillion miles before ever being absorbed by something.
