Can not read more than 4 sec on Linux

[lumag]

Debian Linux, python 3.9.1, rtl sdr dongle by rtl-sdr blog, V3

$ python3 cali.py -v
{'f': None, 'm': 'dab', 's': 'rtlsdr', 'c': 'all', 'rs': 2048000, 'rg': 20, 'rd': 0, 'nsec': 10, 'graph': False, 'verbose': True}
let's find your SDR's oscillator precision
scanning...
starting mode: dab
Detached kernel driver
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Scanning all channels
Scanning…:   0%|                                    | 0/35 [00:00<?, ?it/s]Reattached kernel driver
Scanning…:   0%|                                    | 0/35 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/lumag/Projects/Ham/CalibrateSDR/cali.py", line 199, in <module>
    main(vars(args))
  File "/home/lumag/Projects/Ham/CalibrateSDR/cali.py", line 74, in main
    cali.utils.scan_one_dab_channel(dabchannels, channel, sdr, rs, ns, rg, filename, samplerate,
  File "/home/lumag/Projects/Ham/CalibrateSDR/calibratesdr/utils.py", line 101, in scan_one_dab_channel
    record_with_rtlsdr(sdr, rs, cf, ns, rg, filename)
  File "/home/lumag/Projects/Ham/CalibrateSDR/calibratesdr/utils.py", line 88, in record_with_rtlsdr
    samples = sdr.read_bytes(ns * 2)
  File "/home/lumag/.local/lib/python3.9/site-packages/rtlsdr/rtlsdr.py", line 456, in read_bytes
    raise LibUSBError(result, 'Could not read %d bytes' % (num_bytes))
rtlsdr.rtlsdr.LibUSBError: <LIBUSB_ERROR_NO_MEM (-11): Insufficient memory> "Could not read 40960000 bytes"

[hornig]

Debian Linux, python 3.9.1, rtl sdr dongle by rtl-sdr blog, V3
[...]

  File "/home/lumag/.local/lib/python3.9/site-packages/rtlsdr/rtlsdr.py", line 456, in read_bytes
    raise LibUSBError(result, 'Could not read %d bytes' % (num_bytes))
rtlsdr.rtlsdr.LibUSBError: <LIBUSB_ERROR_NO_MEM (-11): Insufficient memory> "Could not read 40960000 bytes"

Hi @lumag , thanks for the feedback.
can you please check if you got a file "tmp.dat" in your CalibrateSDR folder?
if this is there, the recordning works at least.

the last "memory" notice seems like you do not have enough spare memory. I use Windows and if I have a simular notice, I either have too many things running in parallel, or the file is just too big for Python (and I forgot to use the 64bit version, which can handle more ram) or numpy. but with -nsec = 10 seconds the files are not near 2gb ram.

so first, check if you have a tmp.dat file so I know recording works (I use windows and did not provide any libs for win or linux).
then try again CalibrateSDR with -nsec 4 then the tmp.dat file will be smaller.
maybe tell me what system you use (pc, raspi, etc).

speak you soon :)

[lumag]

There is no tmp.dat.

-nsec 5 fails with the similar message:

rtlsdr.rtlsdr.LibUSBError: <LIBUSB_ERROR_NO_MEM (-11): Insufficient memory> "Could not read 20480000 bytes"

-nsec 4 works, but seems to give no calibration:

{'f': None, 'm': 'dab', 's': 'rtlsdr', 'c': 'all', 'rs': 2048000, 'rg': 20, 'rd': 0, 'nsec': 4.0, 'graph': False, 'verbose': True}
let's find your SDR's oscillator precision
scanning...
starting mode: dab
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Scanning all channels
Scanning…: 100%|███████████████████████████| 35/35 [04:32<00:00,  7.79s/it]

____Results_______________________________________________________________________________________
#   , block, freq [Hz], SNR [dB] , Prec. [ppm], offset [Hz], block [x][o][ ] & signal strength
--------------------------------------------------------------------------------------------------
#  0, 5A   , 174928000,  +0.99264,       None,        +0.0, [ ]  [###############     ] 73%
#  1, 5B   , 176640000,  +1.07592,       None,        +0.0, [ ]  [################    ] 79%
#  2, 5C   , 178352000,  +1.02170,       None,        +0.0, [ ]  [###############     ] 75%
#  3, 5D   , 180064000,  +1.09491,       None,        +0.0, [ ]  [################    ] 80%
#  4, 6A   , 181936000,  +1.36087,       None,        +0.0, [ ]  [####################] 100%
#  5, 6B   , 183648000,  +0.94552,       None,        +0.0, [ ]  [##############      ] 69%
#  6, 6C   , 185360000,  +0.77546,       None,        +0.0, [ ]  [###########         ] 57%
#  7, 6D   , 187072000,  +0.75930,       None,        +0.0, [ ]  [###########         ] 56%
#  8, 7A   , 188928000,  +0.88496,       None,        +0.0, [ ]  [#############       ] 65%
#  9, 7B   , 190640000,  +0.96256,       None,        +0.0, [ ]  [##############      ] 71%
# 10, 7C   , 192352000,  +0.88146,       None,        +0.0, [ ]  [#############       ] 65%
# 11, 7D   , 194064000,  +0.90515,       None,        +0.0, [ ]  [#############       ] 67%
# 12, 8A   , 195936000,  +0.85149,       None,        +0.0, [ ]  [#############       ] 63%
# 13, 8B   , 197648000,  +0.83504,       None,        +0.0, [ ]  [############        ] 61%
# 14, 8C   , 199360000,  +0.72051,       None,        +0.0, [ ]  [###########         ] 53%
# 15, 8D   , 201072000,  +0.56573,       None,        +0.0, [ ]  [########            ] 42%
# 16, 9A   , 202928000,  +0.58411,       None,        +0.0, [ ]  [#########           ] 43%
# 17, 9B   , 204640000,  +0.56797,       None,        +0.0, [ ]  [########            ] 42%
# 18, 9C   , 206352000,  +0.63058,       None,        +0.0, [ ]  [#########           ] 46%
# 19, 9D   , 208064000,  +0.68284,       None,        +0.0, [ ]  [##########          ] 50%
# 20, 10A  , 209936000,  +0.60197,       None,        +0.0, [ ]  [#########           ] 44%
# 21, 10N  , 210096000,  +0.61475,       None,        +0.0, [ ]  [#########           ] 45%
# 22, 10B  , 211648000,  +0.51649,       None,        +0.0, [ ]  [########            ] 38%
# 23, 10C  , 213360000,  +0.58776,       None,        +0.0, [ ]  [#########           ] 43%
# 24, 10D  , 215072000,  +0.62344,       None,        +0.0, [ ]  [#########           ] 46%
# 25, 11A  , 216928000,  +0.61269,       None,        +0.0, [ ]  [#########           ] 45%
# 26, 11N  , 217088000,  +0.61241,       None,        +0.0, [ ]  [#########           ] 45%
# 27, 11B  , 218640000,  +0.60204,       None,        +0.0, [ ]  [#########           ] 44%
# 28, 11C  , 220352000,  +0.62399,       None,        +0.0, [ ]  [#########           ] 46%
# 29, 11D  , 222064000,  +0.58510,       None,        +0.0, [ ]  [#########           ] 43%
# 30, 12A  , 223936000,  +0.59538,       None,        +0.0, [ ]  [#########           ] 44%
# 31, 12N  , 224096000,  +0.56769,       None,        +0.0, [ ]  [########            ] 42%
# 32, 12B  , 225648000,  +0.60087,       None,        +0.0, [ ]  [#########           ] 44%
# 33, 12C  , 227360000,  +0.54024,       None,        +0.0, [ ]  [########            ] 40%
# 34, 12D  , 229072000,  +0.58276,       None,        +0.0, [ ]  [#########           ] 43%

[lumag]

I'm using a laptop, 15 GiB of mem, 4.4 GiB free at this moment.

[OK2AMA]

Executing the script after running without a parameter in my case crashing too, with same error.

With parameter -nsec 4 it works fine:
python3 cali.py -nsec 4

script output:

...
# 31, 12N  , 224096000,  +0.42769,       None,        +0.0, [ ]  [#                   ] 4%
# 32, 12B  , 225648000,  +0.44650,       None,        +0.0, [ ]  [#                   ] 4%
# 33, 12C  , 227360000,  -2.64297,   -57.8529,    -13153.4, [ ]  [                    ] -23%
# 34, 12D  , 229072000, +11.54215,   -57.7258,    -13223.4, [x]  [####################] 100%

@lumag - your text output looks like you have not present DAB signal on RTL SDR receiver input - do you have DAB+ signal coverage in your area? (you can check it by same HW - DAB+ RTL SDR receiver https://github.com/AlbrechtL/welle.io )
You also have significantly higher signal strength on all channels, however, I used a not very suitable antenna for this quick test.

[quantenProjects]

I can confirm the <LIBUSB_ERROR_NO_MEM (-11): Insufficient memory> "Could not read 40960000 bytes" with -nsec over 4.086 (fails with 4.087 with 16740351 bytes). It seems like libusb under linux is not able to allocate enough memory for 5 sec of recording.

But with the a low -nsec it works like a charm!

{'f': None, 'm': 'dab', 's': 'rtlsdr', 'c': '2', 'rs': 2048000, 'rg': 20, 'rd': 0, 'nsec': 4.086, 'graph': False, 'verbose': False}
let's find your SDR's oscillator precision
scanning...
starting mode: dab
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Scanning only channel # 2

____Results_______________________________________________________________________________________
#   , block, freq [Hz], SNR [dB] , Prec. [ppm], offset [Hz], block [x][o][ ] & signal strength
--------------------------------------------------------------------------------------------------
#  2, 5C   , 178352000,  +4.74037,    -5.4584,      -973.5, [x]  [####################] 100%

Archlinux python 3.9.0 pyrtlsdr==0.2.92

[nocarryr]

record_with_rtlsdr in utils.py is using sdr.read_bytes() which returns the low-level ctypes.Array from librtlsdr. Garbage collection should handle things as expected with this under most cases, but record_with_rtlsdr is returning the buffer and it's being thrown away when the function is called

It may be worth trying to either copy the samples to a list before returning it:

_samples = sdr.read_bytes(ns * 2)
samples = _samples[:]
del _samples

or just not returning the array at all

[quantenProjects]

Thanks @nocarryr for looking into this!

I have tried both your suggestions, but the error remains. I isolated the sdr.read_bytes() call and it still throws the error:

https://gist.github.com/quantenProjects/2f42c8455a88b72bd4ccc8b2c3b375ed

$ python -V
Python 3.9.0
$ pip freeze 
cycler==0.10.0
kiwisolver==1.3.1
matplotlib==3.3.3
numpy==1.19.5
Pillow==8.1.0
pyparsing==2.4.7
pyrtlsdr==0.2.92
python-dateutil==2.8.1
scipy==1.6.0
six==1.15.0
tqdm==4.55.1
$ uname -a
Linux buzzbaton 5.9.13-arch1-1 #1 SMP PREEMPT Tue, 08 Dec 2020 12:09:55 +0000 x86_64 GNU/Linux
$ python debug_issue_1.py 
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Traceback (most recent call last):
  File "…CalibrateSDR/misc/debug_issue_1.py", line 8, in <module>
    sdr.read_bytes(20480000)
  File "…CalibrateSDR/venv/lib/python3.9/site-packages/rtlsdr/rtlsdr.py", line 456, in read_bytes
    raise LibUSBError(result, 'Could not read %d bytes' % (num_bytes))
rtlsdr.rtlsdr.LibUSBError: <LIBUSB_ERROR_IO (-1): Input/output error> "Could not read 20480000 bytes"

To me it does not seem like a problem with memory management inside CalibrateSDR (although we should either copy samples or not returning it at all) and more like a problem with rtlsdr or LibUSB. Do you have any other idea?

[quantenProjects]

I have pushed a workaround using read_bytes_async. With it, I'm able to record more than 4 sec.

@OK2AMA and @lumag Can you try out the new version?

@nocarryr I think it would still be interesting, why read_bytes fails with Linux (@hornig has no problem with his windows)