When the remote control has not been used for a few seconds, most of its functions are powered down to conserve energy, including the trackpad, which means it will not react to user input.
In this powered down state the remote control relies on the low power accelerometer for notification of user activity. When the remote control is picked up or nudged lightly, it will resume normal operation.