`vecx` and `vecy` are two mutually perpendicular unit vectors. If the vectors `ahatx + ahaty + c(hatx xx haty), hatx + (hatx xx haty)` and `chatx + chaty +b (hatx xx haty)`, lie in a plane then c is: (A) a,b,c are in AP (B) a,b,c are in GP (C) a,b,c are in HP (D) a,c,b are in GP

Options

A. AM of a and b
B. GM of a and b
C. HM of a and b
D. Equal to zero

Answer

To solve the problem, we need to determine the relationship between the constants \( a \), \( b \), and \( c \) given that the vectors \( \vec{u} \), \( \vec{v} \), and \( \vec{w} \) lie in a plane.

### Step-by-Step Solution:

1. **Identify the Vectors**:
We have three vectors defined as:
- \( \vec{u} = a \hat{x} + a \hat{y} + c (\hat{x} \times \hat{y}) \)
- \( \vec{v} = \hat{x} + (\hat{x} \times \hat{y}) \)
- \( \vec{w} = c \hat{x} + c \hat{y} + b (\hat{x} \times \hat{y}) \)

Since \( \hat{x} \) and \( \hat{y} \) are mutually perpendicular unit vectors, we know that \( \hat{x} \times \hat{y} = \hat{z} \).

Thus, we can rewrite the vectors as:
- \( \vec{u} = a \hat{x} + a \hat{y} + c \hat{z} \)
- \( \vec{v} = \hat{x} + \hat{z} \)
- \( \vec{w} = c \hat{x} + c \hat{y} + b \hat{z} \)

2. **Condition for Coplanarity**:
The vectors \( \vec{u} \), \( \vec{v} \), and \( \vec{w} \) are coplanar if the scalar triple product (or box product) of these vectors is zero:
\[ [\vec{u}, \vec{v}, \vec{w}] = 0 \]

3. **Set Up the Determinant**:
We can express this condition in terms of a determinant:
\[ \begin{vmatrix} a & a & c \\ 1 & 0 & 1 \\ c & c & b \end{vmatrix} = 0 \]

4. **Calculate the Determinant**:
Expanding the determinant:
\[ = a \begin{vmatrix} 0 & 1 \\ c & b \end{vmatrix} - a \begin{vmatrix} 1 & 1 \\ c & b \end{vmatrix} + c \begin{vmatrix} 1 & 0 \\ c & c \end{vmatrix} \]
\[ = a(0 - c) - a(1 \cdot b - 1 \cdot c) + c(1 \cdot c - 0 \cdot c) \]
\[ = -ac - a(b - c) + c^2 \]
\[ = -ac - ab + ac + c^2 \]
\[ = c^2 - ab \]

5. **Set the Determinant to Zero**:
For the vectors to be coplanar:
\[ c^2 - ab = 0 \]
Thus, we have:
\[ c^2 = ab \]

6. **Conclusion**:
The equation \( c^2 = ab \) indicates that \( c \) is the geometric mean of \( a \) and \( b \). Therefore, \( a, c, b \) are in geometric progression (GP).

### Final Answer:
The correct option is (D) \( a, c, b \) are in GP.