I am no expert by any means, but here are my thoughts. Any part has tolerances, so any combination could possibly create a great or not so good fit. My barrel (only have one right now) had a small headspace problem. This can be easily cured if you reload. Accuracy is fine, and as you are aware of, single shots are compact and handy. Trigger is easy to work on and range from very good to mediocre. Interchangability and possible variety are endless. Overall, it is a good system, although overpriced IMO. Most of the hinge problems are more imagined than real, although they can/do exist. Most barrels are plenty accurate. Again, remember that you are working with several pieces machined at different times all with +/- tolerances. You stand a fair chance of getting a great, tight fit, and also a chance of getting a set up having some slop. You just have to realize the inherent design factors and decide if you are willing to deal with any problems that might arise.
You can buy any off the shelf rifle of any manufacturer and stand the possibility of getting a super accurate one, or one needing some attention to make it work to it's best ability. The single shots with interchangeable barrels just add a few extra built in areas that might need that attention.