a. From figure the gravitational force on mass `2m` at `G` due to mas at `A` is
`F_(1)=G(mxx2m)/(1^(2))=2Gm^(2)` along `GA`
Gravitational force of mass `2m` at `G` due to mass at `B` is
`F_(2)=G(mxx2m)/(1^(2))=2Gm^(2)` along `BG`
Gravitational force on mass `2m` at `G` due to mass at `C` is
`F_(3) G(mxx2m)/(1^(2))=2Gm^(2)` along `GC`
Draw `DE` parallel to `BC` passing through point `G`. Then
`/_EGC=30^@=/_DGB`.
Resolving `vecF_(2)` and `vecF_(3)` into two rectangular components we have
`F_(2)cos30^@` along `GD` and `F_(2)sin30^@` along `GH`
`F_(3)cos30^@` along `GE` and `F_(3)` and `30^@` along `GH`
Here `F_2cos30^@` and `F_3 sin 30^@` are equal in magnitude and acting in opposite directions, and cancel in magnitude and acting opposite directions, and cancel out each other.
The resultant force on mass `2m` at `G` is
`F_(1)-(F_(2)sin30^@=F_(3)sin30^@)`
`=2Gm^(2)-(2Gm^(2)xx1/2+2Gm^(2)xx1/2)`
`=0`
b. When mass at `A` is `2m`, then gravitational force on mass `2m` at `G` due to mass `2m` at `A` is
`F_(1)^(')=G(2mmxx2mm)/(1^(2))=4Gm^(2)` along `GA`
The resultant force on mass `2m` at `G` due to masses at At, `B` and `C` is
`F_(1)^(')=(F_(2)sin30^@+F_(3)sin30^@)`
`=4Gm^(2)-(2Gmxx1/2+2Gm^(2)xx1/2)`
`=2Gm^(2)` along `GA`