To save power, most the functions of the remote control, including the trackpad, are powered down when the remote control has not been used for a few seconds. This means then it will not react to user input.
In this powered-down state, the remote control relies on the low-power accelerometer for notification of user activity. When the remote control is picked up or nudged lightly it will resume normal operation.