Skip to content

Filter step: Option to choose different buffer methods #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/images/constant_buffer.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/extrapolated_buffer.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/reflected_buffer.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
78 changes: 76 additions & 2 deletions docs/nodes/steps/filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@ NaN values are re-inserted in their original places in the output.
> **Required:** `False`
> **Default value:** `0`
>
> Extrapolation buffer. Defines how many frames to add on either side
> DEPRECATED: Use the buffer and bufferType options instead.
> Defines how many frames to add on either side
> of the series, useful if the filter handles the edges of the series
> strangely.
>
Expand All @@ -54,6 +55,42 @@ NaN values are re-inserted in their original places in the output.
> is then filled with values linearly extrapolated from these two
> points.
>
> #### `bufferFrames`
>
> **Type:** `Number`
> **Required:** `False`
> **Default value:** `0`
>
> Defines how many frames to add on either side
> of the series, useful if the filter handles the edges of the series
> strangely.
>
> Leading and trailing NaN values are removed before buffering,
> i.e., buffer begins from the first and last real value.
> NaN values are then re-inserted in the original places for
> the output.
>
> #### `bufferType`
>
> **Type:** `String`
> **Required:** `False`
> **Allowed values:** `constant | extrapolate | reflect`
> **Default value:** `extrapolate`
>
> If set to `constant`, the signal is buffered by extending each end with its first and last values, respectively.
> ![Constant buffer](../../images/constant_buffer.png)
>
> If set to `extrapolate`, buffering is made by looking at the first and second values,
> and the last and second-to-last values, respectively. The buffer
> is then filled with values linearly extrapolated from these two
> points.
> ![Extrapolated buffer](../../images/extrapolated_buffer.png)
>
> If set to `reflect`, buffering is done by taking the last number of frames specified in `bufferFrames`,
> creating an inverted copy, reversing this series, and then appending it to the original sequence
> to create a reflection.
> ![Reflected buffer](../../images/reflected_buffer.png)
>
> #### `iterations`
>
> **Type:** `Number`
Expand Down Expand Up @@ -121,7 +158,8 @@ Runs a Butterworth IIR high-pass filter over the input data.
> **Required:** `False`
> **Default value:** `0`
>
> Extrapolation buffer. Defines how many frames to add on either side
> DEPRECATED: Use the buffer and bufferType options instead.
> Defines how many frames to add on either side
> of the series, useful if the filter handles the edges of the series
> strangely.
>
Expand All @@ -135,6 +173,42 @@ Runs a Butterworth IIR high-pass filter over the input data.
> is then filled with values linearly extrapolated from these two
> points.
>
> #### `bufferFrames`
>
> **Type:** `Number`
> **Required:** `False`
> **Default value:** `0`
>
> Defines how many frames to add on either side
> of the series, useful if the filter handles the edges of the series
> strangely.
>
> Leading and trailing NaN values are removed before buffering,
> i.e., buffer begins from the first and last real value.
> NaN values are then re-inserted in the original places for
> the output.
>
> #### `bufferType`
>
> **Type:** `String`
> **Required:** `False`
> **Allowed values:** `constant | extrapolate | reflect`
> **Default value:** `extrapolate`
>
> If set to `constant`, the signal is buffered by extending each end with its first and last values, respectively.
> ![Constant buffer](../../images/constant_buffer.png)
>
> If set to `extrapolate`, buffering is made by looking at the first and second values,
> and the last and second-to-last values, respectively. The buffer
> is then filled with values linearly extrapolated from these two
> points.
> ![Extrapolated buffer](../../images/extrapolated_buffer.png)
>
> If set to `reflect`, buffering is done by taking the last number of frames specified in `bufferFrames`,
> creating an inverted copy, reversing this series, and then appending it to the original sequence
> to create a reflection.
> ![Reflected buffer](../../images/reflected_buffer.png)
>
> #### `iterations`
>
> **Type:** `Number`
Expand Down
48 changes: 35 additions & 13 deletions src/lib/processing/algorithms/filter.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,41 @@ import test from 'ava';

import { f32, mockStep } from '../../../test-utils/mock-step';
import { Signal } from '../../models/signal';
import { SeriesBufferMethod } from '../../utils/series';

import { BaseFilterStep, FilterType, HighPassFilterStep, LowPassFilterStep } from './filter';

const f1 = new Signal(32);
const s1 = new Signal(f32(2, 1, 1, 1, 2, 1, 1, 1, 2), 300);

test('BaseFilterStep - "extrapolate" option', async(t) => {
t.is(mockStep(BaseFilterStep, [f1]).extrapolate, 0); // Default value
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 500 }).extrapolate, 500);
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 5000 }).extrapolate, 1000); // Beyond max value
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: -10 }).extrapolate, 0); // Below max value
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 10.5 }).extrapolate, 10); // should be integer
test('BaseFilterStep - "buffer type" option', async(t) => {
// Test deprecated 'extrapolate' option
t.is(mockStep(BaseFilterStep, [f1]).bufferFrames, 0); // Default value
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 500 }).bufferFrames, 500);
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 5000 }).bufferFrames, 1000); // Beyond max value
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: -10 }).bufferFrames, 0); // Below max value
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 10.5 }).bufferFrames, 10); // should be integer

t.is(mockStep(BaseFilterStep, [f1]).bufferMethod, SeriesBufferMethod.Extrapolate);
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 500 }).bufferMethod, SeriesBufferMethod.Extrapolate);
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 5000 }).bufferMethod, SeriesBufferMethod.Extrapolate);
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: -10 }).bufferMethod, SeriesBufferMethod.Extrapolate);
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 10.5 }).bufferMethod, SeriesBufferMethod.Extrapolate);

// Test bufferFrames and bufferType options
t.is(mockStep(BaseFilterStep, [f1]).bufferFrames, 0); // Default value
t.is(mockStep(BaseFilterStep, [f1], { bufferFrames: 500 }).bufferFrames, 500);
t.is(mockStep(BaseFilterStep, [f1], { bufferFrames: 5000 }).bufferFrames, 1000); // Beyond max value
t.is(mockStep(BaseFilterStep, [f1], { bufferFrames: -10 }).bufferFrames, 0); // Below max value
t.is(mockStep(BaseFilterStep, [f1], { bufferFrames: 10.5 }).bufferFrames, 10); // Should be integer

await t.throws(() => mockStep(BaseFilterStep, [f1], { bufferType: 'undefined' })); // Invalid buffer type
t.is(mockStep(BaseFilterStep, [f1]).bufferMethod, SeriesBufferMethod.Extrapolate);
t.is(mockStep(BaseFilterStep, [f1], { bufferType: 'extrapolate' }).bufferMethod, SeriesBufferMethod.Extrapolate);
t.is(mockStep(BaseFilterStep, [f1], { bufferType: 'reflect' }).bufferMethod, SeriesBufferMethod.Reflect);
t.is(mockStep(BaseFilterStep, [f1], { bufferType: 'constant' }).bufferMethod, SeriesBufferMethod.Constant);
t.is(mockStep(BaseFilterStep, [f1], { extrapolate: 500, bufferFrames: 500, bufferType: 'reflect' }).bufferMethod, SeriesBufferMethod.Reflect); // Prefer bufferType over extrapolate option

});

test('BaseFilterStep - "iterations" option', async(t) => {
Expand Down Expand Up @@ -41,7 +64,7 @@ test('BaseFilterStep - "order" option', async(t) => {
});

test('LowPassFilterStep - basic test', async(t) => {
const step = mockStep(LowPassFilterStep, [s1], { extrapolate: 2, cutoff: 15, iterations: 2, order: 1 });
const step = mockStep(LowPassFilterStep, [s1], { bufferFrames: 2, bufferType: 'extrapolate', cutoff: 15, iterations: 2, order: 1 });

t.is(step.filterType, FilterType.LowPass);

Expand All @@ -60,9 +83,8 @@ test('LowPassFilterStep - basic test', async(t) => {
));
});


test('HighPassFilterStep - basic test', async(t) => {
const step = mockStep(HighPassFilterStep, [s1], { extrapolate: 100, cutoff: 40, iterations: 5, order: 3 });
const step = mockStep(HighPassFilterStep, [s1], { bufferFrames: 100, bufferType: 'extrapolate', cutoff: 40, iterations: 5, order: 3 });

t.is(step.filterType, FilterType.HighPass);

Expand All @@ -83,7 +105,7 @@ test('HighPassFilterStep - basic test', async(t) => {

test('LowPassFilterStep - handle NaNs', async(t) => {
const series = new Signal(f32(undefined, undefined, undefined, 2, 1, 1, 1, 2, 1, 1, 1, 2, undefined, undefined, undefined), 300);
const step = mockStep(LowPassFilterStep, [series], { extrapolate: 2, cutoff: 15, iterations: 2, order: 1 });
const step = mockStep(LowPassFilterStep, [series], { bufferFrames: 2, bufferType: 'extrapolate', cutoff: 15, iterations: 2, order: 1 });

t.is(step.filterType, FilterType.LowPass);

Expand All @@ -109,7 +131,7 @@ test('LowPassFilterStep - handle NaNs', async(t) => {

test('LowPassFilterStep - handle NaNs (no trailing)', async(t) => {
const series = new Signal(f32(undefined, undefined, undefined, 2, 1, 1, 1, 2, 1, 1, 1, 2), 300);
const step = mockStep(LowPassFilterStep, [series], { extrapolate: 2, cutoff: 15, iterations: 2, order: 1 });
const step = mockStep(LowPassFilterStep, [series], { bufferFrames: 2, bufferType: 'extrapolate', cutoff: 15, iterations: 2, order: 1 });

t.is(step.filterType, FilterType.LowPass);

Expand All @@ -132,7 +154,7 @@ test('LowPassFilterStep - handle NaNs (no trailing)', async(t) => {

test('LowPassFilterStep - handle NaNs (no leading)', async(t) => {
const series = new Signal(f32(2, 1, 1, 1, 2, 1, 1, 1, 2, undefined, undefined, undefined), 300);
const step = mockStep(LowPassFilterStep, [series], { extrapolate: 2, cutoff: 15, iterations: 2, order: 1 });
const step = mockStep(LowPassFilterStep, [series], { bufferFrames: 2, bufferType: 'extrapolate', cutoff: 15, iterations: 2, order: 1 });

t.is(step.filterType, FilterType.LowPass);

Expand All @@ -155,7 +177,7 @@ test('LowPassFilterStep - handle NaNs (no leading)', async(t) => {

test('LowPassFilterStep - all NaNs', async(t) => {
const series = new Signal(f32(undefined, undefined, undefined, undefined, undefined, undefined), 300);
const step = mockStep(LowPassFilterStep, [series], { extrapolate: 2, cutoff: 15, iterations: 2, order: 1 });
const step = mockStep(LowPassFilterStep, [series], { bufferFrames: 2, bufferType: 'extrapolate', cutoff: 15, iterations: 2, order: 1 });

const res = await step.process();

Expand Down
122 changes: 108 additions & 14 deletions src/lib/processing/algorithms/filter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@ import * as Fili from 'fili/dist/fili.min.js';
import { PropertyType } from '../../models/property';
import { StepCategory, StepClass } from '../../step-registry';
import { NumberUtil } from '../../utils/number';
import { ProcessingError } from '../../utils/processing-error';
import { SeriesBufferMethod, SeriesUtil } from '../../utils/series';
import { markdownFmt } from '../../utils/template-literal-tags';
import { TypeCheck } from '../../utils/type-check';

import { BaseAlgorithmStep } from './base-algorithm';

Expand Down Expand Up @@ -39,7 +41,8 @@ export type IirFilter = {
`,
})
export class BaseFilterStep extends BaseAlgorithmStep {
extrapolate: number;
bufferFrames: number;
bufferMethod: SeriesBufferMethod;
iterations: number;
cutoffFreq: number;
order: number;
Expand All @@ -53,16 +56,31 @@ export class BaseFilterStep extends BaseAlgorithmStep {
// Use the signal frame rate as the sampling frequency of the filter function.
const sampleFreq = this.inputs[0]?.frameRate || 300;
const characteristic = 'butterworth';

// Parse input parameters
const bufferTypeInput = this.getPropertyValue<SeriesBufferMethod>('bufferType', PropertyType.String, false);
this.bufferMethod = bufferTypeInput || SeriesBufferMethod.Extrapolate;
this.bufferFrames = this.getPropertyValue<number>('bufferFrames', PropertyType.Number, false) || 0;

if (!TypeCheck.isValidEnumValue(SeriesBufferMethod, this.bufferMethod)) {
throw new ProcessingError('Unexpected type in bufferType option.');
}

/* DEPRECATED - Extrapolation as an option is deprecated.
* We keep this for backwards compatibility.
* If the extrapolate option is set, we set the bufferFrames to the number input and the bufferMethod to Extrapolate.
* */
const extrapolate = this.getPropertyValue<number>('extrapolate', PropertyType.Number, false) || 0;
if (extrapolate && !bufferTypeInput) {
this.bufferFrames = extrapolate;
this.bufferMethod = SeriesBufferMethod.Extrapolate;
}

// Extrapolation - how much to add on either side of the series
// useful if the filter handles the edges of the series strangely.
// How much to add on either side of the series useful if the filter handles the edges of the series strangely.
// Valid range: 0 - 1000, default: 0
this.extrapolate = this.getPropertyValue<number>('extrapolate', PropertyType.Number, false) || 0;
this.extrapolate = Math.floor(this.extrapolate);
this.extrapolate = Math.min(this.extrapolate, 1000);
this.extrapolate = Math.max(this.extrapolate, 0);
this.bufferFrames = Math.floor(this.bufferFrames);
this.bufferFrames = Math.min(this.bufferFrames, 1000);
this.bufferFrames = Math.max(this.bufferFrames, 0);

// Iterations - how many times to apply the filter.
// Valid range: 1 - 10, default: 1
Expand Down Expand Up @@ -127,8 +145,8 @@ export class BaseFilterStep extends BaseAlgorithmStep {
let values = [...fixedSeries];

// Add buffer in the beginning and end.
if (this.extrapolate) {
values = SeriesUtil.buffer(values, this.extrapolate, SeriesBufferMethod.Extrapolate);
if (this.bufferFrames) {
values = SeriesUtil.buffer(values, this.bufferFrames, this.bufferMethod);
}

// Apply filter repeatedly according to the iterations property.
Expand All @@ -137,8 +155,8 @@ export class BaseFilterStep extends BaseAlgorithmStep {
}

// Remove buffer from beginning and end.
if (this.extrapolate) {
values = values.slice(this.extrapolate, values.length - this.extrapolate);
if (this.bufferFrames) {
values = values.slice(this.bufferFrames, values.length - this.bufferFrames);
}

// Add the removed leading and trailing NaNs
Expand Down Expand Up @@ -168,7 +186,8 @@ export class BaseFilterStep extends BaseAlgorithmStep {
required: false,
default: '0',
description: markdownFmt`
Extrapolation buffer. Defines how many frames to add on either side
DEPRECATED: Use the buffer and bufferType options instead.
Defines how many frames to add on either side
of the series, useful if the filter handles the edges of the series
strangely.

Expand All @@ -182,6 +201,43 @@ export class BaseFilterStep extends BaseAlgorithmStep {
is then filled with values linearly extrapolated from these two
points.
`,
}, {
name: 'bufferFrames',
type: 'Number',
typeComment: '(min: 0, max: 1000)',
required: false,
default: '0',
description: markdownFmt`
Defines how many frames to add on either side
of the series, useful if the filter handles the edges of the series
strangely.

Leading and trailing NaN values are removed before buffering,
i.e., buffer begins from the first and last real value.
NaN values are then re-inserted in the original places for
the output.
`,
}, {
name: 'bufferType',
type: 'String',
enum: ['constant', 'extrapolate', 'reflect'],
required: false,
default: 'extrapolate',
description: markdownFmt`
If set to ''constant'', the signal is buffered by extending each end with its first and last values, respectively.
![Constant buffer](../../images/constant_buffer.png)

If set to ''extrapolate'', buffering is made by looking at the first and second values,
and the last and second-to-last values, respectively. The buffer
is then filled with values linearly extrapolated from these two
points.
![Extrapolated buffer](../../images/extrapolated_buffer.png)

If set to ''reflect'', buffering is done by taking the last number of frames specified in ''bufferFrames'',
creating an inverted copy, reversing this series, and then appending it to the original sequence
to create a reflection.
![Reflected buffer](../../images/reflected_buffer.png)
`,
}, {
name: 'iterations',
type: 'Number',
Expand Down Expand Up @@ -243,7 +299,8 @@ export class LowPassFilterStep extends BaseFilterStep {
required: false,
default: '0',
description: markdownFmt`
Extrapolation buffer. Defines how many frames to add on either side
DEPRECATED: Use the buffer and bufferType options instead.
Defines how many frames to add on either side
of the series, useful if the filter handles the edges of the series
strangely.

Expand All @@ -257,6 +314,43 @@ export class LowPassFilterStep extends BaseFilterStep {
is then filled with values linearly extrapolated from these two
points.
`,
}, {
name: 'bufferFrames',
type: 'Number',
typeComment: '(min: 0, max: 1000)',
required: false,
default: '0',
description: markdownFmt`
Defines how many frames to add on either side
of the series, useful if the filter handles the edges of the series
strangely.

Leading and trailing NaN values are removed before buffering,
i.e., buffer begins from the first and last real value.
NaN values are then re-inserted in the original places for
the output.
`,
}, {
name: 'bufferType',
type: 'String',
enum: ['constant', 'extrapolate', 'reflect'],
required: false,
default: 'extrapolate',
description: markdownFmt`
If set to ''constant'', the signal is buffered by extending each end with its first and last values, respectively.
![Constant buffer](../../images/constant_buffer.png)

If set to ''extrapolate'', buffering is made by looking at the first and second values,
and the last and second-to-last values, respectively. The buffer
is then filled with values linearly extrapolated from these two
points.
![Extrapolated buffer](../../images/extrapolated_buffer.png)

If set to ''reflect'', buffering is done by taking the last number of frames specified in ''bufferFrames'',
creating an inverted copy, reversing this series, and then appending it to the original sequence
to create a reflection.
![Reflected buffer](../../images/reflected_buffer.png)
`,
}, {
name: 'iterations',
type: 'Number',
Expand Down
Loading