Skip to content

Video/audio sync doesn't work #5359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
prielka opened this issue Jan 25, 2024 · 2 comments · May be fixed by #7376
Open

Video/audio sync doesn't work #5359

prielka opened this issue Jan 25, 2024 · 2 comments · May be fixed by #7376

Comments

@prielka
Copy link

prielka commented Jan 25, 2024

I am trying to do Video Timeline Segmentation in LSF, I have an issue with the audio not syncing to the video, I can play each of them separately but each play button doesn't start both video and audio together.
This is the config I am using:

 <View>
            <Header>Video timeline segmentation</Header>
            <Video name="video" value="$video_url" sync="audio"/>
            <Labels name="videoLabels" toName="audio" choice="multiple">
                <Label value="Moving" />
                <Label value="Not moving" />
            </Labels>
            <Audio name="audio" value="$video_url" sync="video"/>
 </View>

I tried with several different videos, all have the same issue.
I also tried different config formats, with different decoder/player combinations for the audio.
Here is an example video I tried to use: https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/VolkswagenGTIReview.mp4

My LS version is 1.10.1, I am running it on Windows 11, using Chrome.
This is a warning that I see in the console logs:

Waveform.js:320 The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo.gl/7K7WLu
value @ wavesurfer.js:5820
i @ wavesurfer.js:5718
(anonymous) @ wavesurfer.js:1846
i @ wavesurfer.js:1832
value @ wavesurfer.js:4390
value @ wavesurfer.js:4174
value @ wavesurfer.js:5582
componentDidMount @ Waveform.js:320
vl @ react-dom.production.min.js:220
Ts @ react-dom.production.min.js:259
t.unstable_runWithPriority @ scheduler.production.min.js:18
Fo @ react-dom.production.min.js:122
Ms @ react-dom.production.min.js:252
bs @ react-dom.production.min.js:243
ps @ react-dom.production.min.js:237
$s @ react-dom.production.min.js:284
(anonymous) @ react-dom.production.min.js:289
xs @ react-dom.production.min.js:244
iu @ react-dom.production.min.js:289
t.render @ react-dom.production.min.js:296
createApp @ LabelStudio.js:62
await in createApp (async)
qoe @ LabelStudio.js:31
(anonymous) @ LabelStudioComponent.js:13
commitHookEffectListMount @ react-dom.development.js:23150
commitPassiveMountOnFiber @ react-dom.development.js:24926
commitPassiveMountEffects_complete @ react-dom.development.js:24891
commitPassiveMountEffects_begin @ react-dom.development.js:24878
commitPassiveMountEffects @ react-dom.development.js:24866
flushPassiveEffectsImpl @ react-dom.development.js:27039
flushPassiveEffects @ react-dom.development.js:26984
(anonymous) @ react-dom.development.js:26769
workLoop @ scheduler.development.js:266
flushWork @ scheduler.development.js:239
performWorkUntilDeadline @ scheduler.development.js:533

I don't have any browser extensions active.
@sandy1990418
Copy link

sandy1990418 commented Jun 27, 2024 •
edited
Loading

I faced the same problem and resolved it by separating the video and audio elements. Here's the my solution:

<View>
    <Video name="video" value="$video" sync="audio"/>
    <Audio name="audio" value="$audio" sync="video"/>    
    <Labels name="label" toName="audio">
      <Label value="Speech"/>
    </Labels>
    <TextArea name="transcription" toName="audio" editable="true" perRegion="true" required="true" maxSubmissions="1" rows="5" />
</View>

@prielka
Copy link
Author

prielka commented Aug 11, 2024

I tried this and it did not work unfortunately

cloudmark added a commit to cloudmark/label-studio that referenced this issue Apr 15, 2025
@cloudmark
feat: Add sync between TimeSeries, Video, Audio, and other TimeSeries…
808c4a5 
… components

Fixes #4892: How can we label video and time series signals while watching them together?
Fixes HumanSignal#5359: Video/audio sync doesn't work

- Add sync attribute to TimeSeries tag model to enable group-based synchronization
- Compose TimeSeries with SyncableMixin for event broadcasting and handling
- Implement sync event handlers for seek, play, and pause events
- Add smooth view updates during audio/video playback
- Prevent event flooding by suppressing sync events during playback
- Maintain view window size during playback
- Map media timeline to TimeSeries range (start/end points)

This enables synchronized playback and seeking between Video, Audio, and TimeSeries
components when they share the same sync group. Multiple TimeSeries components can
also sync with each other, allowing for coordinated view updates across different
time series data. For example:

```
<View>
  <Video name="video" value="$video" sync="group_a"/>
  <AudioPlus name="audio" value="$audio" sync="group_a"/>
  <TimeSeries name="accel_timeseries" value="$accel_data" sync="group_a">
    <Channel column="accel_x" strokeColor="#FF0000"/>
    <Channel column="accel_y" strokeColor="#00FF00"/>
  </TimeSeries>
  <TimeSeries name="gyro_timeseries" value="$gyro_data" sync="group_a">
    <Channel column="gyro_x" strokeColor="#0000FF"/>
    <Channel column="gyro_y" strokeColor="#FF00FF"/>
  </TimeSeries>
</View>
```
@cloudmark cloudmark linked a pull request Apr 15, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Add sync between TimeSeries, Video, Audio cloudmark/label-studio
2 participants
@sandy1990418 @prielka