The unit vector which is orthogonal to the vector `5hati + 2hatj + 6hatk ` and is coplanar with vectors `2hati + hatj + hatk and hati - hatj + hatk ` is (a) `(2hati - 6hatj + hatk)/sqrt41` (b) `(2hati-3hatj)/sqrt13` (c) `(3 hatj -hatk)/sqrt10` (d) `(4hati + 3hatj - 3hatk)/sqrt34`

Options

A. `(2hati - 6hatj + hatk)/sqrt41`
B. `(2hati-3hatj)/sqrt13`
C. `(3 hati -hatk)/sqrt10`
D. `(4hati + 3hatj - 3hatk)/sqrt34`

Explanation

To find the unit vector that is orthogonal to the vector \( \mathbf{A} = 5\hat{i} + 2\hat{j} + 6\hat{k} \) and is coplanar with the vectors \( \mathbf{B} = 2\hat{i} + \hat{j} + \hat{k} \) and \( \mathbf{C} = \hat{i} - \hat{j} + \hat{k} \), we can follow these steps:

### Step 1: Express the required vector
Let the required vector be \( \mathbf{D} = x\hat{i} + y\hat{j} + z\hat{k} \). Since \( \mathbf{D} \) is coplanar with \( \mathbf{B} \) and \( \mathbf{C} \), we can express \( \mathbf{D} \) as a linear combination of \( \mathbf{B} \) and \( \mathbf{C} \):
\[ \mathbf{D} = \lambda \mathbf{B} + \mu \mathbf{C} \]
Substituting the vectors:
\[ \mathbf{D} = \lambda (2\hat{i} + \hat{j} + \hat{k}) + \mu (\hat{i} - \hat{j} + \hat{k}) \]
This expands to:
\[ \mathbf{D} = (2\lambda + \mu)\hat{i} + (\lambda - \mu)\hat{j} + (\lambda + \mu)\hat{k} \]

### Step 2: Set up the orthogonality condition
Since \( \mathbf{D} \) is orthogonal to \( \mathbf{A} \), we have:
\[ \mathbf{A} \cdot \mathbf{D} = 0 \]
Calculating the dot product:
\[ (5\hat{i} + 2\hat{j} + 6\hat{k}) \cdot ((2\lambda + \mu)\hat{i} + (\lambda - \mu)\hat{j} + (\lambda + \mu)\hat{k}) = 0 \]
This gives:
\[ 5(2\lambda + \mu) + 2(\lambda - \mu) + 6(\lambda + \mu) = 0 \]

### Step 3: Simplify the equation
Expanding the equation:
\[ 10\lambda + 5\mu + 2\lambda - 2\mu + 6\lambda + 6\mu = 0 \]
Combining like terms:
\[ (10\lambda + 2\lambda + 6\lambda) + (5\mu - 2\mu + 6\mu) = 0 \]
This simplifies to:
\[ 18\lambda + 9\mu = 0 \]
From this, we can express \( \mu \) in terms of \( \lambda \):
\[ \mu = -2\lambda \]

### Step 4: Substitute back into the expression for \( \mathbf{D} \)
Substituting \( \mu = -2\lambda \) into the expression for \( \mathbf{D} \):
\[ \mathbf{D} = (2\lambda - 2\lambda)\hat{i} + (\lambda + 2\lambda)\hat{j} + (\lambda - 2\lambda)\hat{k} \]
This simplifies to:
\[ \mathbf{D} = 0\hat{i} + 3\lambda\hat{j} - \lambda\hat{k} \]
Thus:
\[ \mathbf{D} = 3\lambda\hat{j} - \lambda\hat{k} \]

### Step 5: Find the magnitude of \( \mathbf{D} \)
The magnitude of \( \mathbf{D} \) is:
\[ |\mathbf{D}| = \sqrt{(3\lambda)^2 + (-\lambda)^2} = \sqrt{9\lambda^2 + \lambda^2} = \sqrt{10\lambda^2} = |\lambda|\sqrt{10} \]

### Step 6: Find the unit vector
The unit vector \( \hat{D} \) in the direction of \( \mathbf{D} \) is given by:
\[ \hat{D} = \frac{\mathbf{D}}{|\mathbf{D}|} = \frac{3\lambda\hat{j} - \lambda\hat{k}}{|\lambda|\sqrt{10}} = \frac{3\hat{j} - \hat{k}}{\sqrt{10}} \]

### Conclusion
Thus, the required unit vector is:
\[ \hat{D} = \frac{3\hat{j} - \hat{k}}{\sqrt{10}} \]

### Final Answer
The correct option is (c) \( \frac{3\hat{j} - \hat{k}}{\sqrt{10}} \).