If `a!=b`, then the system of equations `ax+by+bz=0, bx+ay+bz=0, bx+by+az=0` will have a non-trivial solution if (1) `a+b=0` (2) `a+2b=0` (3) `2a+b=0` (4) `a+4b=0`

Explanation

For a system of linear equations having non-trivial solution, value of the determinant should be `0`.

Here, `Delta = |[a,b,b],[b,a,b],[b,b,a]| = 0`

Applying `R_1->R_1+R_2+R_3`

`|[a+2b,a+2b,a+2b],[b,a,b],[b,b,a]| = 0`

`=>(a+2b)|[1,1,1],[b,a,b],[b,b,a]| = 0`

Applying `C_2->C_2-C_1 and C_3->C_3-C_1`

`=>(a+2b)|[1,0,0],[b,a-b,0],[b,0,a-b]| = 0`

`=>(a+2b)[1((a-b)^2-0)] = 0`

...