I have a 1911Colt Gold Cup with a NM Slide, Barrel and Bushing. This gun is built specifically for SWC loads and is commonly known in the Standard Outdoor Pistol competetion as a "Wad Gun".
I load 3.8 grains of Bullseye to push a 200 grain SWC bullet. I get about 1" groups at 25 yards with this combination and the brass just drops to my right about a foot away from where I'm standing. But understand that this is a heavy slide (sight Rib added to the top of the slide for added weight) pistol made to perform with these light loads.
The only problem I see with the light loads is the complete cycling of the slide to cock the pistol and load another round. Less than 3.2 grains of Bullseye (200 grain SWC bullets) and I do not have reliable cycling of the slide. If you are "Focused on the front sight" like you're supposed to be, you can actually see some of the rounds going down range. 3.4 grains of Bullseye pushing a 185 grain SWC gives me reliable cycling of the slide.
With the SWC bullets, you have to have a widened feed ramp on the barrel and magazines that have been reworked to feed the SWC bullets reliably.
If you plan on using a standard 1911A1 series pistol, you have to have a lighter recoil spring and if you're shooting SWC bullets, you may have to have the feed ramp on the barrel reworked and widened. JBPMidas has a standard 1911A1 style pistol that shoots reliably using 4.0 grains of Bullseye with a 185 grain SWC Hornady bullet.
YES, you can damage your pistol by going way too light. When you get to the point that the pistol will not function reliably, you need to stop and start back up with your powder charge or make changes to your recoil spring. When the pistol doesn't function on every round, that should tell you to STOP and do something different.