Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LTC SMPTE: Local freerunning clock running behind #287

Closed
dirkdamerau opened this issue Dec 3, 2024 · 18 comments
Closed

LTC SMPTE: Local freerunning clock running behind #287

dirkdamerau opened this issue Dec 3, 2024 · 18 comments
Assignees

Comments

@dirkdamerau
Copy link

During more testing of the NTP to LTC functionality I noticed that after an NTP sync (and of course every reboot) the LTC is in sync to other GPS locked clocks but it seem to be running in freerun until the next NTP sync in about 15-20 minutes. During that time of ~17 minutes (in my case) it is already running 2 seconds behind the GPS clock. With the new NTP sync it than jumps back to correct time. e.g. from 19:52:33 to 19:52:35 as well as 20:09:38 to 20:09:40 during my tests.

I also noticed some frame miss alignments in the LTC output every ~37s. e.g. [... :18,:19,:20,:21,:22,:21,:24,:00 ...] Letting the OPi run for roughly 12 hours freeruning (disconnected NTP Server) the time difference was about 3 minutes. That would be around 1f every 10s.

Any ideas how this could be fixed? Thanks Dirk

@vanvught
Copy link
Owner

vanvught commented Dec 3, 2024

Hi Dirk,

I will try to replicate the free running clock problem that you are experiencing.

The free running clock is based on H3_TIMER->AVS_CNT0

const uint32_t timer = H3_TIMER->AVS_CNT0; // Millis timer

According to the datasheet, there is a more accurate high speed timer. I will investigate of using this timer will improve the free running clock.

Thanks, Arjan

Scherm­afbeelding

@dirkdamerau
Copy link
Author

dirkdamerau commented Dec 3, 2024

Hi Arjan, did some further testing. The issue does not happen when using source "internal". It is freerunning now for more than 30 minutes and has no visible lag to the GPS clock running next to it. Also the ~37s frame miss alignment seams not to happen. So I guess it is something just in combination with NTP Client.

UPDATE: After more than 12 hours the "internal" clock is running less than 1s ahead. That sounds reasonable for a free running clock.

@vanvught vanvught changed the title Local freerunning clock running behind LTC SMPTE: Local freerunning clock running behind Dec 4, 2024
@vanvught
Copy link
Owner

vanvught commented Dec 4, 2024

Note: Changed NTP polling from

static constexpr uint8_t POLL_POWER = 3;

to
# define CONFIG_NTP_CLIENT_POLL_POWER 10

@dirkdamerau
Copy link
Author

Do you want me to test anything ? Where can I find the new Image ?

@vanvught
Copy link
Owner

vanvught commented Dec 4, 2024

Hi Dirk,

I have reverted the NTP Client poll power back to 3. This would set the system time quicker, hence less drift.
I am investigation using the high speed timer as well.

Thanks, Arjan

@vanvught
Copy link
Owner

vanvught commented Dec 5, 2024

orangepi_zero.uImage.gz

Latest build.

@dirkdamerau
Copy link
Author

Thank you. At first the lates build looked quiet good. Viewing the clocks by eye they seem to be and stay perfectly in sync over pretty long time. But as my timecode reader has noticed some drifts in framerate I took a deeper look. I recorded the LTC output and analyzed it with ltcdump from this project https://github.com/x42/ltc-tools. This analyzes the LTC audio file and detects wrong frame as well as the aligment to the audio samples.

Here is an example on what happend several times in similar ways:

#User bits Timecode | Pos. (samples)
00000000 19:36:16:20 | 74217343 74219262
00000000 19:36:16:21 | 74219263 74221182
00000000 19:36:16:22 | 74221183 74223102
00000000 19:36:16:23 | 74223103 74225022
00000000 19:36:16:24 | 74225023 74226942
00000000 19:36:17:00 | 74226943 74228862
00000000 19:36:17:01 | 74228863 74230782
#DISCONTINUITY
00000000 19:36:17:00 | 74230783 74232702
#DISCONTINUITY
00000000 19:36:17:02 | 74232703 74234622
#DISCONTINUITY
00000000 19:36:17:04 | 74234623 74236542
00000000 19:36:17:05 | 74236543 74238462
00000000 19:36:17:06 | 74238463 74240381
#DISCONTINUITY
00000000 19:36:17:05 | 74240382 74242301
#DISCONTINUITY
00000000 19:36:17:07 | 74242302 74244220
#DISCONTINUITY
00000000 19:36:17:09 | 74244221 74246141
00000000 19:36:17:10 | 74246142 74248061
00000000 19:36:17:11 | 74248062 74249981
#DISCONTINUITY
00000000 19:36:17:10 | 74249982 74251901
#DISCONTINUITY
00000000 19:36:17:12 | 74251902 74253821
#DISCONTINUITY
00000000 19:36:17:14 | 74253822 74255741
00000000 19:36:17:15 | 74255742 74257661
00000000 19:36:17:16 | 74257662 74259581
#DISCONTINUITY
00000000 19:36:17:15 | 74259582 74261501
#DISCONTINUITY
00000000 19:36:17:17 | 74261502 74263421
#DISCONTINUITY
00000000 19:36:17:19 | 74263422 74265341
00000000 19:36:17:20 | 74265342 74267261
00000000 19:36:17:21 | 74267262 74269181
#DISCONTINUITY
00000000 19:36:17:20 | 74269182 74271100
#DISCONTINUITY
00000000 19:36:17:22 | 74271101 74273020
#DISCONTINUITY
00000000 19:36:17:24 | 74273021 74274940
#DISCONTINUITY
00000000 19:36:17:22 | 74274941 74276860
#DISCONTINUITY
00000000 19:36:18:00 | 74276861 74278780
00000000 19:36:18:01 | 74278781 74280700
00000000 19:36:18:02 | 74280701 74282620
00000000 19:36:18:03 | 74282621 74284540
00000000 19:36:18:04 | 74284541 74286460
00000000 19:36:18:05 | 74286461 74288380
00000000 19:36:18:06 | 74288381 74290300
00000000 19:36:18:07 | 74290301 74292220
00000000 19:36:18:08 | 74292221 74294140
00000000 19:36:18:09 | 74294141 74296060
00000000 19:36:18:10 | 74296061 74297980
00000000 19:36:18:11 | 74297981 74299900
00000000 19:36:18:12 | 74299901 74301819
00000000 19:36:18:13 | 74301820 74303739
00000000 19:36:18:14 | 74303740 74305659
00000000 19:36:18:15 | 74305660 74307579
00000000 19:36:18:16 | 74307580 74309499
00000000 19:36:18:17 | 74309500 74311419
00000000 19:36:18:18 | 74311420 743133

Here is another example where the clock really stops for a short moment, even visually on the small OLED screen:

00000000 19:36:29:15 | 74808684 74810603
00000000 19:36:29:16 | 74810604 74812523
00000000 19:36:29:17 | 74812524 74814443
00000000 19:36:29:18 | 74814444 74816363
00000000 19:36:29:19 | 74816364 74818283
00000000 19:36:29:20 | 74818284 74820203
00000000 19:36:29:21 | 74820204 74822122
00000000 19:36:29:22 | 74822123 74824039
00000000 19:36:29:23 | 74824040 74825962
00000000 19:36:29:24 | 74825963 74827882
#DISCONTINUITY
00000000 19:36:29:23 | 74827883 74829802
00000000 19:36:29:24 | 74829803 74831722
#DISCONTINUITY
00000000 19:36:29:24 | 74831723 74833642
#DISCONTINUITY
00000000 19:36:29:24 | 74833643 74835562
#DISCONTINUITY
00000000 19:36:29:24 | 74835563 74837482
#DISCONTINUITY
00000000 19:36:29:24 | 74837483 74839402
#DISCONTINUITY
00000000 19:36:29:24 | 74839403 74841322
#DISCONTINUITY
00000000 19:36:29:24 | 74841323 74843242
#DISCONTINUITY
00000000 19:36:29:24 | 74843243 74845162
#DISCONTINUITY
00000000 19:36:29:24 | 74845163 74847082
#DISCONTINUITY
00000000 19:36:29:24 | 74847083 74849002
#DISCONTINUITY
00000000 19:36:29:24 | 74849003 74850922
#DISCONTINUITY
00000000 19:36:29:24 | 74850923 74852841
00000000 19:36:30:00 | 74852842 74854758
00000000 19:36:30:01 | 74854759 74856681
00000000 19:36:30:02 | 74856682 74858601
00000000 19:36:30:03 | 74858602 74860521
00000000 19:36:30:04 | 74860522 74862441
00000000 19:36:30:05 | 74862442 74864361

Interesting facts I noticed:

  • If errors occur they always appear close to xx:xx:xx:00 (last or first frame of a second)
  • Sometimes these errors occur up to every 8 seconds
  • Sometimes they don't appear for 30 min at all
  • They also appear when using "internal" source - but very very less often (see picture below)

I put these ltcdump data to an excel file and printed errors over time (vertical line = some error):
1x 90min with NTP source
1x 140min with Internal source

Bildschirmfoto 2024-12-05 um 22 37 29

Following issue has probably the same reason:
#288

vanvught added a commit that referenced this issue Dec 6, 2024
vanvught added a commit that referenced this issue Dec 6, 2024
@vanvught
Copy link
Owner

vanvught commented Dec 7, 2024

Hi Dirk,

Sometimes these errors occur up to every 8 seconds

That is 2^3 seconds were the NTP Client is synchronising the local system time.

I am working on changing the timer for the local system time. However this needs testing HDMI on the OPi One, and DMX on both boards.

Thanks, Arjan

@vanvught
Copy link
Owner

vanvught commented Dec 7, 2024

References to HS_TIMER

  • ./lib-h3/src/h3_udelay.cpp: nTicksPrevious = H3_HS_TIMER->CURNT_LO;
  • ./lib-h3/src/h3_udelay.cpp: const auto nTicksNow = H3_HS_TIMER->CURNT_LO;
  • ./lib-h3/src/irq_timer.cpp: h3_timer0_func(H3_HS_TIMER->CURNT_LO);
  • ./lib-h3/src/irq_timer.cpp: h3_timer1_func(H3_HS_TIMER->CURNT_LO);
  • ./lib-h3/src/h3_hs_timer_init.cpp: H3_HS_TIMER->IRQ_EN = 0; // Disable interrupts
  • ./lib-h3/src/h3_hs_timer_init.cpp: H3_HS_TIMER->CTRL = 0; // Timer Stop/Pause
  • ./lib-h3/src/h3_hs_timer_init.cpp: H3_HS_TIMER->INTV_LO = 0xFFFFFFFF;
  • ./lib-h3/src/h3_hs_timer_init.cpp: H3_HS_TIMER->INTV_HI = 0x00FFFFFF;
  • ./lib-h3/src/h3_hs_timer_init.cpp: H3_HS_TIMER->CTRL |= (0x1 << 4);
  • ./lib-h3/src/h3_hs_timer_init.cpp: H3_HS_TIMER->CTRL |= CTRL_RELOAD;
  • ./lib-h3/src/h3_hs_timer_init.cpp: H3_HS_TIMER->CTRL |= CTRL_START;
  • ./lib-dmx/src/h3/single/dmx.cpp: gsv_RdmDataReceiveEnd = H3_HS_TIMER->CURNT_LO;
  • ./lib-dmx/src/h3/single/dmx.cpp: sv_nFiqMicrosCurrent = H3_HS_TIMER->CURNT_LO;
  • ./lib-dmx/src/h3/single/dmx.cpp: gsv_RdmDataReceiveEnd = H3_HS_TIMER->CURNT_LO;
  • ./lib-dmx/src/h3/single/dmx.cpp: gsv_RdmDataReceiveEnd = H3_HS_TIMER->CURNT_LO;
  • ./lib-dmx/src/h3/multi/dmx.cpp: gsv_RdmDataReceiveEnd = H3_HS_TIMER->CURNT_LO;
  • ./lib-dmx/src/h3/multi/dmx.cpp: gsv_RdmDataReceiveEnd = H3_HS_TIMER->CURNT_LO;
  • ./lib-h3/device/fb/h3_hdmi.cpp: tmo = h3_hs_timer_lo_us() + 2000;
  • ./lib-h3/device/fb/h3_hdmi.cpp: if (h3_hs_timer_lo_us() > tmo) {
  • ./lib-h3/device/fb/h3_hdmi.cpp: start = h3_hs_timer_lo_us();
  • ./lib-h3/device/fb/h3_hdmi.cpp: } while ((h3_hs_timer_lo_us() - start) < 300);
  • ./lib-network/src/esp8266/h3/esp8266.cpp: uint32_t micros_now = h3_hs_timer_lo_us();
  • ./lib-network/src/esp8266/h3/esp8266.cpp: while ((!(PORT_CIN->DAT & (1 << CIN))) && (h3_hs_timer_lo_us() - micros_now < (uint32_t) 500000))

vanvught added a commit that referenced this issue Dec 7, 2024
vanvught added a commit that referenced this issue Dec 7, 2024
@dirkdamerau
Copy link
Author

Let me know if I can test new builds on the OPi Zero. During this process I am still not sure would be the right way to go. If the LTC is updated to some NTP clock from time to time neccessarily there will be jumps in the LTC that would make the time more accurate but would also lead in some missing or some double frames. The other option would be to just update every 24h during the night so errors are minimized but that would need a good freerunning during the day to have at least 1-2 seconds drift in 24h. Perfect solution would be something that aligns the LTC (system time basis) to the NTP permanently by stepping up or down the clock speed according to the reference, so no jumps need to occur in the LTC. The chrony project does on linux for system time to NTP sync but I don't know if that could work on an OPi as well.

@vanvught
Copy link
Owner

vanvught commented Dec 9, 2024

Perfect solution would be something that aligns the LTC (system time basis) to the NTP permanently by stepping up or down the clock speed according to the reference, so no jumps need to occur in the LTC. The chrony project does on linux for system time to NTP sync but I don't know if that could work on an OPi as well.

That should be possible on the OPi Zero as it run Linux as well. I will look into the OPi driver for Linux.

@vanvught
Copy link
Owner

vanvught commented Dec 9, 2024

but that would need a good freerunning during the day to have at least 1-2 seconds drift in 24h

That is what I am working with. As it breaks current code, it is some time expensive work.

@vanvught
Copy link
Owner

vanvught commented Dec 9, 2024

Hi Dirk, please will you give the attached firmware a test?

I have set the NTP Client poll power back to 10. And using now the ARM Generic Timer as done with Linux.

Thanks, Arjan

orangepi_zero.uImage.gz

vanvught added a commit that referenced this issue Dec 10, 2024
@dirkdamerau
Copy link
Author

sorry for the delay ... this release is now syncing every 2^10 seconds to the ntp server but the LTC also jumps + ~15min with every sync. Here are two examples happend right after each other in the same recording.

00000000 16:30:07:00 | 48903307 48905226
00000000 16:30:07:01 | 48905227 48907146
#DISCONTINUITY
00000000 16:45:01:00 | 48907147 48909066
00000000 16:45:01:01 | 48909067 48910986
#DISCONTINUITY
00000000 16:45:02:00 | 48910987 48912906
00000000 16:45:02:01 | 48912907 48914826
.....
00000000 17:02:06:00 | 98059482 98061401
#DISCONTINUITY
00000000 17:02:05:24 | 98061402 98063321
#DISCONTINUITY
00000000 17:20:00:00 | 98063322 98065241
00000000 17:20:00:01 | 98065242 98067161

The clock it self seams to be accureate than the one before. I did not see any noticeable drift during the 17min. But in between the syncs there are still every few minutes lost and sometimes wrong ordered frames in the ltc output.:

00000000 16:29:54:22 | 48321566 48323485
00000000 16:29:54:23 | 48323486 48325405
00000000 16:29:54:24 | 48325406 48327325
#DISCONTINUITY
00000000 16:29:55:01 | 48327326 48329245
00000000 16:29:55:02 | 48329246 48331165
00000000 16:29:55:03 | 48331166 48333085
00000000 16:29:55:04 | 48333086 48335005
00000000 16:29:55:05 | 48335006 48336924
00000000 16:29:55:06 | 48336925 48338844
00000000 16:29:55:07 | 48338845 48340764
00000000 16:29:55:08 | 48340765 48342684
00000000 16:29:55:09 | 48342685 48344604
00000000 16:29:55:10 | 48344605 48346524
00000000 16:29:55:11 | 48346525 48348444
00000000 16:29:55:12 | 48348445 48350364
00000000 16:29:55:13 | 48350365 48352284
00000000 16:29:55:14 | 48352285 48354204
00000000 16:29:55:15 | 48354205 48356124
00000000 16:29:55:16 | 48356125 48358044
00000000 16:29:55:17 | 48358045 48359964
00000000 16:29:55:18 | 48359965 48361884
00000000 16:29:55:19 | 48361885 48363804
00000000 16:29:55:20 | 48363805 48365724
00000000 16:29:55:21 | 48365725 48367644
00000000 16:29:55:22 | 48367645 48369563
00000000 16:29:55:23 | 48369564 48371483
00000000 16:29:55:24 | 48371484 48373403
#DISCONTINUITY
00000000 16:29:55:23 | 48373404 48375323
#DISCONTINUITY
00000000 16:29:56:00 | 48375324 48377243
00000000 16:29:56:01 | 48377244 48379163

@vanvught
Copy link
Owner

vanvught commented Dec 13, 2024

Hi Dirk,

I would like to split this issue in 2 items:

  1. Local freerunning clock running behind
  2. NTP Client update issue (LTC SMPTE: Improve NTP Client synchronise process.  #292)

When you disable the NTP Client, then the local free running is oké ?

Thanks, Arjan

@dirkdamerau
Copy link
Author

dirkdamerau commented Dec 14, 2024

no problem, running on source "internal" it is very accurate and stable now, I've not logged a single error in 6 hours and drift is less than 0,5 seconds (OPi is running faster than real time), also acceptable from my point of view, I'll update my post tomorrow after 24h

UPDATE: ... the internal ltc does not exceed 23:59:59:24
UPDATE after 12h: ... internal ltc is running apprx 1s ahead

@vanvught
Copy link
Owner

UPDATE: ... the internal ltc does not exceed 23:59:59:24

That is as designed. Otherwise the time coded show will start at the beginning 00:00:00:00 again. It is not expected that there is a time code show for more than 24 hours.

@dirkdamerau
Copy link
Author

You are right for timecode shows, I'm thinking of using this as a master time of day that repeats every day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants