The modem trace is captured using the nrfutil trace lte subcommand.
nrfutil device list --traits modem
nrfutil trace lte --input-serialport COM10 --output-pcapng live-trace.pcapng
nrfutil trace lte --input-serialport COM10 --output-wireshark path_to_wireshark