Troubleshooting a Troverlo Tag Reader

References

Console logs

The Tag Readers push out console logs that can help us diagnose issues.

We can get to this information by plugging the USB into a computer and running an SSH program called putty that will both push these logs to the computer screen (dos window) as well as to a file (in the download folder)

Where do I get putty and how do I set it up so it logs to a file?

PuTTY: a free SSH and Telnet client
specifically - latest page - Download PuTTY: release 0.76
Saving Your Putty Sessions To A Log File – Packet6

How do I program putty to see the Tag’s console logs?

Find the COM port that your Tag Reader is using in the Device Manager

Type "device" in Windows search
On the Device manager screen, look under "Ports" for the newest port, once you have plugged in your Tag Reader
In my example below it has COM40 but yours will be different

Launch Putty

Setup the configuration as shown below in the pictures below

(A) Set the Serial line w the Com port used in the batch above or found in the device manager
(B) Set the speed to 115200
(C) Select the Serial Connection type
(D) Type the session name in the “saved sessions” field
(E) Click on "logging" on the left side
(G) Select All sessions output
(H) Enter the path for your downloads folder and enter the file name shown
1. Example is: C:\Users\Mark\Downloads\&H-&Y&M&D-&T.log
2. The “&H-&Y&M&D-&T.log” puts a timestamp on the file, you don’t have to do this, you can call it “putty.log” if you want…advantage is that if you do more than one run, it will not overwrite the previous one.
(I) Click on Session - which will take you back to the above screen
(J) Click on Save
Next time, just select the session name you saved and click (L) Load and then (K) Open

NOTES

You can’t start putty first – it needs the device powered to run.
But….If you can start the putty log “as soon” as you plug in the reader, it will capture important init information that will help me.
So…the best method is to plug in the reader, get putty all configured following the instructions above, make sure the log is created where you think it is and has data…then unplug the reader and close putty
Then plug in the reader, open putty and select the session name you saved and click (L) "Load" and then (K) "Open"

Command line options

These can be done in Putty or in VS (Visual Studio)

To get access to enter a command
1. Ctrl+C to break (this can take several tries...)
2. Anything done after the Ctrl-C is likely done without at least one of the threads running (since you killed some by pressing Ctrl-C). So, that will yield unexpected results.
3. Reboot when you're done!
This clears EVERYTHING - firmware and persistent data (i.e. cached observations), you have to reload the board
1. ```
>>> os.fsformat("/flash")
```
2. there is no feedback after this is done
3. Ctrl-C
4. ```
>>> import config
```
5. Then enter any config parm from the config.py file you want to get the value for
6. Lots of options to get what is configured for that board while it is running:
  
  __class__ __name__ const UART mem32 os pybytes PybytesConfig pybytes_config Pybytes sqnsupgrade hwdt_init HWWATCHDOG_INTERVAL RTC_CNTL_WDTCONFIG0_REG RTC_CNTL_WDTCONFIG1_REG RTC_CNTL_WDTCONFIG2_REG RTC_CNTL_WDTCONFIG3_REG RTC_CNTL_WDTCONFIG4_REG RTC_CNTL_WDTFEED_REG RTC_CNTL_WDTWPROTECT_REG hwdt_feed HW_VERSION PROTO_VERSION MAC_filter RETRANSMISSION_MIN_SPACE OTA_FW_CHECK_INTERVAL WATCHDOG_INTERVAL THREAD_CHECK_INTERVAL HEALTH_CHECK_INTERVAL POST_TIMEOUT_INTERVAL GPS_TIMEOUT VERY_SLOW_speed VERY_SLOW_data_speed VERY_SLOW_obs_per_post SLOW_speed SLOW_data_speed SLOW_obs_per_post FAST_speed FAST_data_speed FAST_obs_per_post VERY_FAST_data_speed VERY_FAST_obs_per_post OTA_SERVER OTA_SERVER_PORT post_url post_auth post_cookie DEBUG_PRINT_POST DEBUG_PRINT_AP_LIST RETRANS_DIR RETRANS_FAIL_THRESH RETRANS_PRINT_COUNT RETRANS_PERSISTENCE_ENABLE USED_SAVED_SSL_SESSION THREAD_SLEEP_OTA RUN_GPS_THREAD USE_NTP NTP_SERVER NTP_UPDATE_PERIOD
To see what OS is running on the board:
1. ```
>>>os.uname()
```
To reboot the processor:
1. ```
>>>import machine
```
2. ```
>>>machine.reset()
```

To see what modem firmware is being used

```
>>>import sqnsupgrade
```
```
>>>sqnsupgrade.info()
```

To reset the modem:
1. if not starting:
  1. ```
  >>>os.fsformat("/flash")  w
```
2. The modem command most likely takes over the modem, which can cause the running threads to become disrupted. Really not supposed to run the sequans utilities with any software running at all. So, that will yield unexpected results. So reboot when you're done!
2. Start VS and connect to Pycom board
3. Unplug power from Reader
4. Plug back in
5. Click the reset button on FIPY ( GPY ) when it starts back up
6. Enter at prompt:
  1. ```
  >>>from network import LTE
```
2. ```
>>>lte = LTE()
```
  3. ```
  >>>lte.reset()
```
4. ```
>>>lte.init()
```
7. To upgrade the firmware on the board, copy the below commands using Ctrl-C:
  1. ```
  from network import LTE
  
  lte = LTE()
  lte.reset()
  
  from machine import SD
  import os
  
  sd = SD()
  os.mount(sd, '/sd')
  
  import sqnsupgrade
```
8. In VSCode, you can right click and it will execute each command in series

NOTE: You cannot easily make changes using Putty. That requires VSCode to upload new firmware

Output logs in Dos console

Example items on the console or in the log to look for:

Timestamp of the beginning of the putty session

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2022.05.03 10:30:23 =~=~=~=~=~=~=~=~=~=~=~=

Firmware version is shown at bootup

TROVERLO READER START: V 1.4

INIT 1.4+USCell+DisGPSOpt+NTP

Where the data is being sent, shown at bootup:

POST URL: https://troverlostagingapi.azurewebsites.net/api/clientapi/V1/groupassetdetections

Tag ID (MAC address) of the reader, shown at bootup

Reader MAC Addr: f0:08:d1:ce:bb:d4

Discovering your Sim card and carrier for LTE-m

AT&T SIM FOUND

Trying to connect to the LTE-m carrier

Observation in 14 secs

attaching.............................Observation in 7 secs

...connnecting

LTE CONNECTED

(NOTE: It may take 10-30 seconds to connect at init, if carrier is not obtained, it will retry several times, then reboot to ensure we clear everything out before it tries again)

Loss of LTE-m carrier connection

LTE LINK DOWN
conn = True
attach = True
post_fail_count = 2
get_post_timer = 300.2847

LTE RE-CONNECT

(NOTE: LTE-M doesn't declare a lost connection very often. In fact, you could be oblivious to the fact that the modem has lost connection. This is a side effect of the way LTE-M works, which is different than LTE)

Good post to the Troverlo Cloud server

Status code= 201

Reason = b'Created'

Encoding = utf-8

Response =

Client problem with post to Troverlo Cloud server

Status code= 400

Reason = b'Bad Request'

Encoding = utf-8

Response = 400.0 BadRequest

Server-side problem with post to Troverlo Cloud server

Status code= 503

Reason = b'Service Unavailable'

Encoding = utf-8

Response = <h1>App Service is offline.</h1>

Health stats - they roll by periodically - to learn more about stats, search for "Health Check" at this link: Theory of Operations for the Troverlo Mobile Tag Reader (TTR-MHr1)

3 different types of resets

rst:0x10 (RTCWDT_RTC_RESET),boot:0x1b (SPI_FAST_FLASH_BOOT)

rst:0x1 (POWERON_RESET),boot:0x16 (SPI_FAST_FLASH_BOOT)

rst:0x7 (TG0WDT_SYS_RESET),boot:0x16 (SPI_FAST_FLASH_BOOT)

rst:0xc (SW_CPU_RESET),boot:0x17 (SPI_FAST_FLASH_BOOT)

There are multiple ways to reset: power cycle, thread monitor reboot, software watchdog, and hardware watchdog. These represent the different methods that the processor was reset. I’d have to double-check, but believe TGO is the SW-commanded reset, which comes from the thread monitor. RTC is the real-time clock. It’s likely the HW watchdog.

What do these mean?

retransmit_index_cleanup index scan complete

/flash/retransmit_buffer/R00011971.ret DOES NOT EXIST. REMOVING.

Remove of /flash/retransmit_buffer/R00011971.ret FAILED

/flash/retransmit_buffer/index0.ret is an index file

/flash/retransmit_buffer/index1.ret is an index file

These are not common, but also not un-expected. When the reader gets reset while it is writing to the flash, it could write a bad file to flash. At boot, the reader scans the file system and does a cleanup. Looks like the file that needs to be deleted would not delete for some reason.

Resetting hardware

There is a button by the LED on the "Hat" that will reset the processor

The main board (pytrack or pysense) just provide power that the FIPY/GPY (the "hat") sits on.

When you pull the power, it resets everything (modem, processor, everything else).

The watchdog and thread monitors can only reset the processor.

We do NOT have a reset in the watchdog for the modem, have to do this manually via command line.