Show that straight lines `(A^2-3b^2)x^2+8A Bx y(b^2-3A^2)y^2=0` form with the line `A x+B y+C=0` an equilateral triangle of area `(C^2)/(sqrt(3(A^2+B^2)))` .

Explanation

Let the line through origin making an angle of `60^(@)` with the line `Ax+By+C=0`be`y=mx`.
`:.tan60^(@)= |(m-(-A//B))/(1+m(-A//B))|`
or `3(B-Am)^(2)=(mB+A)^(2)`
Putting `m=y//x` in above , we get
`(A^(2)-3B^(2))x^(2)+8ABxy+(B^(2)=3A^(2))y^(2)=0`
This is pair of straight lines through origin in which each component line makes `60^(@)` with line `Ax+By+C=0`
Also , length of perpendicular from origin to line `Ax+By+C=0` is
`|C|/(sqrt(A^(2)+B^(2)))`
`:.` Length of side of triangle `(|2C|)/(sqrt(3(A^(2)+B^(2))))`
`:.` Area of triangle `=(sqrt(3))/(4)((2C)/(sqrt(3(A^(2)+B^(2)))))^(2)=(C^(2))/(sqrt(3)(A^(2)+B^(2)))`