An isosceles triangle that can be inscribed in an ellipse `(x^(2))/(a^(2))+(y^(2))/(b^(2))=1` having its vertex coincident with one extremity of major axis has the maximum area equal to `(msqrt(n))/4ab` (`m,n` are prime numbers) then `(m^(2)-n)/3=`

Explanation

To solve the problem, we need to find the maximum area of an isosceles triangle inscribed in the ellipse given by the equation \(\frac{x^2}{a^2} + \frac{y^2}{b^2} = 1\). The vertex of the triangle coincides with one extremity of the major axis of the ellipse.

### Step-by-step Solution:

1. **Identify the coordinates of the ellipse**:
The ellipse has its major axis along the x-axis and minor axis along the y-axis. The vertices of the ellipse are at \((a, 0)\), \((-a, 0)\), \((0, b)\), and \((0, -b)\).

2. **Set the coordinates of the triangle**:
...