If A and B are two matrices of order 3 such that `AB=O` and `A^(2)+B=I`, then tr. `(A^(2)+B^(2))` is equal to ________.

Explanation

To solve the problem step by step, we start with the given equations and properties of matrices.

**Step 1: Write down the given equations.**
We have two matrices \( A \) and \( B \) of order 3 such that:
1. \( AB = O \) (where \( O \) is the zero matrix)
2. \( A^2 + B = I \) (where \( I \) is the identity matrix)

**Step 2: Rearrange the second equation.**
From the second equation, we can express \( B \) in terms of \( A \):
\[ B = I - A^2 \tag{1} \]

**Step 3: Multiply equation (1) by \( B \).**
Now, we will multiply both sides of equation (1) by \( B \):
\[ A^2B + B^2 = B \]

**Step 4: Substitute \( AB = O \) into the equation.**
Since \( AB = O \), we can replace \( A^2B \) with \( O \):
\[ O + B^2 = B \]
This simplifies to:
\[ B^2 = B \tag{2} \]

**Step 5: Substitute \( B \) back into equation (2).**
Using equation (1) in equation (2):
\[ (I - A^2)^2 = I - A^2 \]

**Step 6: Expand \( (I - A^2)^2 \).**
Expanding the left side:
\[ I - 2A^2 + A^4 = I - A^2 \]
This simplifies to:
\[ -2A^2 + A^4 = -A^2 \]
Rearranging gives:
\[ A^4 - A^2 = 0 \]
Factoring out \( A^2 \):
\[ A^2(A^2 - I) = 0 \]

**Step 7: Analyze the equation.**
This implies that either \( A^2 = 0 \) or \( A^2 = I \). Since \( A \) is a 3x3 matrix, we consider the case where \( A^2 = 0 \) (the zero matrix).

**Step 8: Substitute back to find \( B \).**
If \( A^2 = 0 \), then from equation (1):
\[ B = I - A^2 = I - 0 = I \]

**Step 9: Calculate \( A^2 + B^2 \).**
Now we can calculate \( A^2 + B^2 \):
\[ A^2 + B^2 = 0 + I^2 = 0 + I = I \]

**Step 10: Find the trace.**
The trace of the identity matrix \( I \) of order 3 is:
\[ \text{tr}(I) = 3 \]

Thus, the final answer is:
\[ \text{tr}(A^2 + B^2) = 3 \]

---