Skip to content

Stable Config file: target system properties in process_arguments and support template variables in YamlParser #8690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
May 6, 2025

Conversation

mtoffl01
Copy link
Contributor

@mtoffl01 mtoffl01 commented Apr 10, 2025

What Does This Do

This PR adds support for matching system property keys in the Stable Configuration selector, even when those properties include values. For example, a selector with -Darg1 will now correctly match a Java process started with -Darg1=value.

It also introduces support for template variables in the Stable Configuration file schema. These variables can resolve values from process arguments and environment variables. (See reference for more details.)

Notes & Limitations:

Scope Limitation: This change restricts process_arguments selectors and template variables to System Properties only.
Required Format: System Properties must be referenced using their full -D prefix, without specifying a value (e.g., use -Dkey, not key nor -Dkey=value).

Motivation

Stable Config users must be able to target services based on the presence of a specific System Property, regardless of its value. For instance, -Darg1 should match any process started with -Darg1=abcd, -Darg1=xyz, etc.

Currently, our only known use case for process_arguments selectors involves System Properties. If a need arises to support additional types of JVM arguments in the future, we can revisit and expand this functionality.

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@pr-commenter
Copy link

pr-commenter bot commented Apr 10, 2025

Benchmarks

Startup

Load

Dacapo

@mtoffl01 mtoffl01 changed the title [WIP] Parse system properties into key-vals in CLIHelper Apr 10, 2025
@mtoffl01 mtoffl01 changed the title Parse system properties into key-vals in CLIHelper Split system properties into key-vals in CLIHelper Apr 10, 2025
@mtoffl01 mtoffl01 changed the title Split system properties into key-vals in CLIHelper Split system properties into key-vals in CLIHelper and support template variables in YamlParser Apr 16, 2025
@mtoffl01 mtoffl01 marked this pull request as ready for review April 17, 2025 20:47
@mtoffl01 mtoffl01 requested a review from a team as a code owner April 17, 2025 20:47
@mtoffl01 mtoffl01 requested a review from smola April 17, 2025 20:47
Copy link
Contributor

github-actions bot commented Apr 17, 2025

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@smola smola requested review from a team and dougqh and removed request for a team and smola April 23, 2025 06:38
Copy link
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @mtoffl01

I feel there is a mix of concerns about how the CLIHelper changed.
As it’s a common composent, we would expect him to keep providing the JVM arguments. (-Dkey=value being a process argument in itself after all)

While I think that parsing JVM properties is interesting (and needed), it should not replace the process arguments API.
So what about having:

  • one way to retrieve arguments as a list of string (as before)
  • one way of querying JVM properties (get the key value if any)
  • but still do the parsing in one pass

Overall, I also think the common component needs:

  • A complete Javadoc to define its expected capability / behavior,
  • Some tests in order to check edge case behavior (like property redefinition, empty property value, etc...).
    What do you thing about it?

And my mid-term goal is to make this component evolves to a system / env component that will be provide any kind of information about the execution context (JDK version, vendor, native-image, etc...).

@mtoffl01 mtoffl01 changed the title Split system properties into key-vals in CLIHelper and support template variables in YamlParser Support system property key-vals in CLIHelper and support template variables in YamlParser May 1, 2025
@mtoffl01 mtoffl01 changed the title Support system property key-vals in CLIHelper and support template variables in YamlParser Target system properties in process_arguments and support template variables in YamlParser May 1, 2025
@mtoffl01 mtoffl01 changed the title Target system properties in process_arguments and support template variables in YamlParser Stable Config file: target system properties in process_arguments and support template variables in YamlParser May 1, 2025
throw new IOException("Empty environment variable name in template");
}
String value = System.getenv(envVar.toUpperCase());
if (value == null || value.isEmpty()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question - will users ever want to substitute a variable with an empty string?

Currently we don't support this, because the variable will be replaced with UNDEFINED - but a user might want to have some kind of optional variable that may have a value or optionally be the empty string...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressing in a separate PR: #8759

Copy link
Contributor

@mcculls mcculls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a question about whether we should allow users to set variables to the empty string or not

Copy link
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would avoid creating one-liner helpers in internal-api.
They can be replaced by their equivalent static method calls from JDK.

Comment on lines 9 to 12
static writeFileRaw(Path filePath, String data) {
StandardOpenOption[] openOpts = [StandardOpenOption.WRITE] as StandardOpenOption[]
Files.write(filePath, data.getBytes(), openOpts)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that useful to create a dedicated method for a single call which can already be written: Files.write(path, data.getBytes())?
Default open options looks good enough for the testing, aren't they?
And I would not recommend re-using this method as it hide the encoding / charset to used (which is usually not recommended).

Comment on lines 15 to 21
try {
return Files.createTempFile("testFile_", ".yaml")
} catch (IOException e) {
println "Error creating file: ${e.message}"
e.printStackTrace()
return null
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, this is one-liner helper.
Checking the exception is not required is groovy so you can directly write Files.createTempFile() in your test, and the exception handling won't be useful as you will certainly end up with another exception (NullPoienterException) if something fails.

Additionally, Spock and test framework usually provide tools for temporary files (for example @TempDir for Spock: https://spockframework.org/spock/docs/2.3/all_in_one.html#_temp_dir)

@mtoffl01 mtoffl01 enabled auto-merge (squash) May 5, 2025 16:34
@mtoffl01 mtoffl01 merged commit 7169626 into master May 6, 2025
454 checks passed
@mtoffl01 mtoffl01 deleted the mtoff/scfg_fix branch May 6, 2025 09:13
@github-actions github-actions bot added this to the 1.49.0 milestone May 6, 2025
svc-squareup-copybara pushed a commit to cashapp/misk that referenced this pull request May 7, 2025
| Package | Type | Package file | Manager | Update | Change |
|---|---|---|---|---|---|
|
[com.google.cloud:google-cloud-logging](https://github.com/googleapis/java-logging)
| dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`3.22.2` -> `3.22.3` |
|
[com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore)
| dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.28.0` -> `2.28.1` |
| [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java)
| dependencies | misk/gradle/libs.versions.toml | gradle | minor |
`1.48.2` -> `1.49.0` |
| [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.31.36` -> `2.31.37` |
| [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.31.36` -> `2.31.37` |
|
[software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava)
| dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.31.36` -> `2.31.37` |
| [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.31.36` -> `2.31.37` |
| [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.31.36` -> `2.31.37` |
| [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.31.36` -> `2.31.37` |
| [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.31.36` -> `2.31.37` |

---

### Release Notes

<details>
<summary>googleapis/java-logging
(com.google.cloud:google-cloud-logging)</summary>

###
[`v3.22.3`](https://github.com/googleapis/java-logging/blob/HEAD/CHANGELOG.md#3223-2025-05-06)

##### Bug Fixes

- **deps:** Update the Java code generator (gapic-generator-java) to
2.56.3
([844f4fa](googleapis/java-logging@844f4fa))

##### Dependencies

- Update dependency com.google.cloud:sdk-platform-java-config to v3.46.3
([#&#8203;1801](googleapis/java-logging#1801))
([d7aa7bc](googleapis/java-logging@d7aa7bc))
- Update dependency com.google.cloud:sdk-platform-java-config to v3.47.0
([#&#8203;1803](googleapis/java-logging#1803))
([5967ffe](googleapis/java-logging@5967ffe))
- Update googleapis/sdk-platform-java action to v2.57.0
([#&#8203;1804](googleapis/java-logging#1804))
([e9a27ec](googleapis/java-logging@e9a27ec))

</details>

<details>
<summary>googleapis/java-datastore
(com.google.cloud:google-cloud-datastore)</summary>

###
[`v2.28.1`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2281-2025-05-06)

##### Dependencies

- Update dependency com.google.cloud:sdk-platform-java-config to v3.47.0
([#&#8203;1841](googleapis/java-datastore#1841))
([ac393e6](googleapis/java-datastore@ac393e6))
- Update googleapis/sdk-platform-java action to v2.57.0
([#&#8203;1842](googleapis/java-datastore#1842))
([0745906](googleapis/java-datastore@0745906))

</details>

<details>
<summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary>

###
[`v1.49.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.49.0):
1.49.0

### Components

#### Configuration at Runtime

- ✨ Add process tags as list to remote config payload
([#&#8203;8705](DataDog/dd-trace-java#8705) -
[@&#8203;amarziali](https://github.com/amarziali))

#### Continuous Integration Visibility

- 🐛 Add span propagation for Pekko scheduled tasks
([#&#8203;8765](DataDog/dd-trace-java#8765) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- ✨ Update test.retry_reason to use full name of the feature
([#&#8203;8689](DataDog/dd-trace-java#8689) -
[@&#8203;daniel-mohedano](https://github.com/daniel-mohedano))
- 🧹 Remove unused TestEventsHandler methods
([#&#8203;8674](DataDog/dd-trace-java#8674) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))

#### Dynamic Instrumentation

- 🐛 Fix exclude identifiers normalization
([#&#8203;8742](DataDog/dd-trace-java#8742) -
[@&#8203;jpbempel](https://github.com/jpbempel))
- ✨ Make source file tracking asynchronous
([#&#8203;8684](DataDog/dd-trace-java#8684) -
[@&#8203;jpbempel](https://github.com/jpbempel))
- ✨ Add scope filtering for symbol extraction
([#&#8203;8676](DataDog/dd-trace-java#8676) -
[@&#8203;jpbempel](https://github.com/jpbempel))
- ✨ Add support for [@&#8203;key](https://github.com/key) and
[@&#8203;value](https://github.com/value) for Map filtering
([#&#8203;8669](DataDog/dd-trace-java#8669) -
[@&#8203;jpbempel](https://github.com/jpbempel))

#### Library Injection

- ✨ Add system property to force injection of the tracing
library even though multiple javaagents have been detected
([#&#8203;8697](DataDog/dd-trace-java#8697) -
[@&#8203;cecile75](https://github.com/cecile75))

#### Metrics

- ✨ Allow dogstatsd port to be configurable with
DD_DOGSTATSD_PORT
([#&#8203;8693](DataDog/dd-trace-java#8693) -
[@&#8203;randomanderson](https://github.com/randomanderson))

#### Profiling

- ✨ Bump ddprof-java to 1.25.1
([#&#8203;8750](DataDog/dd-trace-java#8750) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- 🐛 Remove cleanup-on-shutdown for temporary files
([#&#8203;8746](DataDog/dd-trace-java#8746) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- ✨⚡ Replace a regex-based SMAP parser with a hand-crafted
one
([#&#8203;8730](DataDog/dd-trace-java#8730) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- ✨ Improve error reporting on profiler startup
([#&#8203;8714](DataDog/dd-trace-java#8714) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- ✨ Exclude ProxyLeakTask exception from exception profiling
([#&#8203;8666](DataDog/dd-trace-java#8666) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- ✨ Use jvmstat for JDKs 9+ programmatically
([#&#8203;8641](DataDog/dd-trace-java#8641) -
[@&#8203;MattAlp](https://github.com/MattAlp))

#### Telemetry

- ✨ Allow dogstatsd port to be configurable with
DD_DOGSTATSD_PORT
([#&#8203;8693](DataDog/dd-trace-java#8693) -
[@&#8203;randomanderson](https://github.com/randomanderson))
- 🐛 Fix appsec.waf.requests telemetry metric
([#&#8203;8644](DataDog/dd-trace-java#8644) -
[@&#8203;jandro996](https://github.com/jandro996))

#### Tracer core

- ✨ Exclude jackson afterburner dynamic classes from
instrumentation
([#&#8203;8747](DataDog/dd-trace-java#8747) -
[@&#8203;amarziali](https://github.com/amarziali))
- ✨ Introduce Java 8 bytecode bridge for instrumentation API
([#&#8203;8736](DataDog/dd-trace-java#8736) -
[@&#8203;PerfectSlayer](https://github.com/PerfectSlayer))
- ⚡🧹 Use byte-buddy classes optimized for Java8+
([#&#8203;8735](DataDog/dd-trace-java#8735) -
[@&#8203;mcculls](https://github.com/mcculls))
- 🐛 Do not set the hibernate or datanucleus span service name when
disabled
([#&#8203;8727](DataDog/dd-trace-java#8727) -
[@&#8203;ygree](https://github.com/ygree))
- ✨ Update bytebuddy and ASM to support JDK 24
([#&#8203;8720](DataDog/dd-trace-java#8720) -
[@&#8203;sarahchen6](https://github.com/sarahchen6))
- 🐛 Turn off JDK socket support by default
([#&#8203;8715](DataDog/dd-trace-java#8715) -
[@&#8203;mcculls](https://github.com/mcculls))
- 🐛 Log warning when trace buffer overflow occurs
([#&#8203;8712](DataDog/dd-trace-java#8712) -
[@&#8203;ygree](https://github.com/ygree))
- ✨🧪 Introducing an internal integration name
([#&#8203;8708](DataDog/dd-trace-java#8708) -
[@&#8203;amarziali](https://github.com/amarziali))
- ✨ Add process tags to client stats payload
([#&#8203;8704](DataDog/dd-trace-java#8704) -
[@&#8203;amarziali](https://github.com/amarziali))
- ✨ Collect process tags for tracing
([#&#8203;8698](DataDog/dd-trace-java#8698) -
[@&#8203;amarziali](https://github.com/amarziali))
- ✨ Stable Config file: target system properties in
process_arguments and support template variables in YamlParser
([#&#8203;8690](DataDog/dd-trace-java#8690) -
[@&#8203;mtoffl01](https://github.com/mtoffl01))
- ✨⚡ Use prefix trie for proxy ignores
([#&#8203;8678](DataDog/dd-trace-java#8678) -
[@&#8203;amarziali](https://github.com/amarziali))
- ✨ Allow agent to be automatically injected when running aside
Log4J patch agent
([#&#8203;8648](DataDog/dd-trace-java#8648) -
[@&#8203;paullegranddc](https://github.com/paullegranddc))
- ✨ Use jvmstat for JDKs 9+ programmatically
([#&#8203;8641](DataDog/dd-trace-java#8641) -
[@&#8203;MattAlp](https://github.com/MattAlp))

#### Tracer internal logging

- 🐛 Delete print line
([#&#8203;8686](DataDog/dd-trace-java#8686) -
[@&#8203;sarahchen6](https://github.com/sarahchen6))

### Instrumentations

#### Akka instrumentation

- 🐛 Handle reentrant scope cleanup in Akka/Pekko actor
instrumentations
([#&#8203;8722](DataDog/dd-trace-java#8722) -
[@&#8203;mcculls](https://github.com/mcculls))

#### Apache Spark instrumentation

- ✨ Use OpenLineage root parent information to generate trace
id ([#&#8203;8726](DataDog/dd-trace-java#8726)
- [@&#8203;mobuchowski](https://github.com/mobuchowski))
- ✨ Spark job cancellation no longer marks application as
failed
([#&#8203;8701](DataDog/dd-trace-java#8701) -
[@&#8203;paul-laffon-dd](https://github.com/paul-laffon-dd))

#### JDBC instrumentation

- 💡 Add support for sybase tds jdbc driver
([#&#8203;8764](DataDog/dd-trace-java#8764) -
[@&#8203;amarziali](https://github.com/amarziali))

#### Kotlin instrumentation

- 🐛 Take defensive copy of parent scope stack when closing nested
coroutines
([#&#8203;8749](DataDog/dd-trace-java#8749) -
[@&#8203;mcculls](https://github.com/mcculls))

#### Reactor instrumentation

- ✨⚡ Do not inspect reactor context when not needed
([#&#8203;8745](DataDog/dd-trace-java#8745) -
[@&#8203;amarziali](https://github.com/amarziali))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am
every weekday" in timezone Australia/Melbourne, Automerge - At any time
(no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Never, or you tick the rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config help](https://github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Renovate
Bot](https://github.com/renovatebot/renovate).

GitOrigin-RevId: 795f347ae34d056efc1194c2f606cee7bca1beea
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants