The straight lines
`(A^(2)-3B^(2))x^(2)+8ABXy+(B^(2)-3A^(2))y^(2)=0` form with the line `Ax+By+C=0` an equilateral triangle of area

Options

A. `(c^(2))/(sqrt(2).(A^(2)+B^(2)))`
B. `(c^(2))/(sqrt(3).(A^(2)+B^(2)))`
C. `(c^(2))/(A^(2)+B^(2))`
D. none of these

Answer

To solve the problem step by step, we will analyze the given equation of the straight lines and the line forming an equilateral triangle. The area of the triangle will be derived from the properties of the lines and the triangle.

### Step 1: Identify the given equations
The given equation of the straight lines is:
\[ (A^2 - 3B^2)x^2 + 8ABxy + (B^2 - 3A^2)y^2 = 0 \]
And the line forming the triangle is:
\[ Ax + By + C = 0 \]

### Step 2: Determine the slopes of the lines
The equation of the straight lines can be rewritten in the form of a quadratic equation in \(y\):
\[ (B^2 - 3A^2)y^2 + 8ABxy + (A^2 - 3B^2)x^2 = 0 \]
Using the quadratic formula, the slopes \(m_1\) and \(m_2\) of the lines can be found from:
\[ m = \frac{-B \pm \sqrt{B^2 - 4AC}}{2A} \]
Here, \(A = B^2 - 3A^2\), \(B = 8AB\), and \(C = A^2 - 3B^2\).

### Step 3: Calculate the slopes
Using the formula for the slopes:
\[ m_1, m_2 = \frac{-8AB \pm \sqrt{(8AB)^2 - 4(B^2 - 3A^2)(A^2 - 3B^2)}}{2(B^2 - 3A^2)} \]
This will yield two slopes \(m_1\) and \(m_2\).

### Step 4: Find the slope of the line \(Ax + By + C = 0\)
The slope of the line \(Ax + By + C = 0\) is given by:
\[ m_3 = -\frac{A}{B} \]

### Step 5: Use the angle condition for the equilateral triangle
For the triangle to be equilateral, the angles between the lines must be \(60^\circ\) and \(120^\circ\). The tangent of the angle between two lines with slopes \(m_1\) and \(m_2\) is given by:
\[ \tan(\theta) = \left|\frac{m_1 - m_2}{1 + m_1 m_2}\right| \]
Setting this equal to \(\tan(60^\circ) = \sqrt{3}\) gives us the condition:
\[ \left|\frac{m_1 - m_2}{1 + m_1 m_2}\right| = \sqrt{3} \]

### Step 6: Solve for the area of the triangle
The area \(A\) of the triangle formed by the lines can be calculated using the formula:
\[ \text{Area} = \frac{1}{2} \times \text{base} \times \text{height} \]
In this case, we can express the area in terms of \(C\) and the slopes derived earlier.

### Step 7: Final expression for area
From the properties of the triangle and the derived slopes, we can express the area as:
\[ \text{Area} = \frac{C^2}{\sqrt{3}(A^2 + B^2)} \]

### Conclusion
Thus, the area of the equilateral triangle formed by the lines is:
\[ \text{Area} = \frac{C^2}{\sqrt{3}(A^2 + B^2)} \]