We solve the situation in the reference frame of centre of mass As only `F` is the external force acting on the system, due to this force, the acceleration of the centre of mass is `F/(M+m)`. Thus, with respect to centre of mass, there is as pseudoforce on the two mases in the opposite direction,the free body digram of `m` and `M` with respect to centre of mass (taking centre of mass at rest) is shown in figure.
Taking centre of mass ast rest, if `m` moves maximum by a distance `x_(1)` and `M` moves maximum by a distance `x_(2)` then the work done by external forces (including pseudoforce) will be
`W=(mF)/(m+M)x_(1)+(F-(MF)/(m+M))x_(2)=(mF)/(m+M)(x_(1)+x_(2))`
this work is stored in the form of potential energy of the spring as `U=(1/2)k(x_(1)+x_(2))^(2)`.
Thus, on equation we get the maximum extension in the spring, as after this instant the sprig starts contracting.
`1/2k(x_(1)+x_(2))^(2)=(mF)/(m+M)(x_(1)+x_(2))`
`implies x_(max)=x_(1)+x_(2)=(2mF)/(k(m+M))`
