Commit 72fa842
authored
[Flink] Make Flink 2.x job submission work on REMOTE mode (#4500)
* [Flink] Fix Flink SQL job submission failures on the client path
Six defects on the path that turns a saved Flink SQL application into a
submitted job. Each was found by submitting a real Flink SQL job to a
standalone cluster and fixing whatever failed next; they are independent
of each other but all sit on this one path.
1. ClassLoaderUtils.runAsClassLoader restored the context classloader
captured in a static field when the class was first initialized —
whichever thread happened to load it — instead of the one the calling
thread had on entry. On pooled threads that silently replaces an
unrelated thread's context classloader.
2. FlinkClientTrait.getCustomCommandLines and
RemoteClient.getStandAloneClusterDescriptor call into Flink classes
bound to the Flink version bundled with this module, while the calling
thread's context classloader is FlinkShimsProxy's target-version shims
classloader. Their internal ServiceLoader lookups therefore resolved
providers from a different Flink version than the interfaces bundled
here, failing with ServiceConfigurationError ("not a subtype"). Both
call sites now run under their own class's classloader. Closes #4483.
3. The build-response getters (workspacePath, pass, shadedJarPath,
flinkBaseImage, mainJarPath, extraLibJarPaths, flinkImageTag,
podTemplatePaths, dockerInnerMainJarPath) do not follow JavaBean
getter naming and carried no @JsonProperty, so Jackson silently
skipped them: every build result persisted to t_flink_app's
buildResultJson lost its paths, and only pass survived — by the
coincidence that its field default is already true. A later submit
then read back shadedJarPath == null and failed with an NPE, an
"entry point class not found", or "flinkJobJar is null", depending on
which downstream path consumed it.
4. SubmitRequest.userJarFile() passed shadedJarPath() straight to
new File(...), which throws NPE when it is legitimately null.
5. streampark-console-service declared a compile dependency on
streampark-flink-shims-base but not on streampark-flink-shims-base-v2,
so FlinkTableInitializerV2 never reached the console's lib/ and every
Flink 2.x SQL job failed with NoClassDefFoundError. Flink 1.x was
unaffected, which is why this went unnoticed.
6. PackagedProgram's setUserClassPaths, disabled wholesale for #3761, is
re-enabled for FLINK_SQL jobs only, so a SQL job's connector jars reach
the client classpath. Verified against a real cluster not to reproduce
the ClassCastException #3761 describes, and it leaves every other job
type on the existing behaviour.
* [Flink] Make Flink 2.x job submission work on REMOTE mode
Submitting a Flink SQL job to a Flink 2.x cluster failed with a
NoClassDefFoundError long before reaching the cluster. Flink 1.x was
unaffected, which is why this went unnoticed. Five independent causes,
each of which only becomes visible after the previous one is fixed:
1. FlinkShimsProxy stopped putting the version-specific shims jar into
the shims classloader. It matches on a name shaped
"streampark-flink-shims_flink-<ver>_<scala>", which those artifacts
carried until e770d2e renamed them without the Scala suffix. Both
spellings are accepted now.
2. The same rename silenced the rule that pulls in the rest of
StreamPark's Flink jars ("has a _<scala> suffix"), so the client stack
was loaded by the console's own classloader and resolved
org.apache.flink.* from the console's fixed baseline Flink instead of
the target version's jars. That is the actual mechanism behind #4483:
the ServiceLoader mismatch it reports is what a half-populated shims
classloader looks like from the outside.
3. shims-base and shims-base-v2 share twelve class names — v2 redeclares
them for Flink 2.x and inherits the rest — but both were added to
every shims classloader regardless of the target version, in directory
listing order. Which Flink version a class had been compiled for was
therefore decided by the filesystem. A 1.x target no longer sees v2 at
all, and a 2.x target gets v2 ahead of the base.
4. SavepointConfigOptions was removed in Flink 2.x, and Configuration's
typed accessors (getBoolean/setBoolean/getInteger over a ConfigOption)
went with it. Since this module is compiled once against a single
baseline but submits to whichever version the user registered, both
are now addressed portably: the savepoint options are declared from
their keys, which are byte-identical across every supported version,
and the generic get/set replace the typed accessors.
5. ClusterClient#submitJob widened its parameter from JobGraph to
ExecutionPlan in 2.x. The instance satisfies either signature, only
the declared type moved, so the call is made reflectively.
A FLINK_SQL program's classloader is also told to resolve
org.apache.streampark.* parent-first. The fat jar bundles whichever shims
it was built against, while the parent is the shims classloader for the
version actually registered; loading both ends in a LinkageError as soon
as one references the other.
Verified against real standalone clusters by driving StreamPark's own
submission path out-of-process: a Flink SQL job now submits and reaches
FINISHED on Flink 2.2.1, and the same job on Flink 1.20.4 — which worked
before this change — still does.
Not addressed: LocalClient and KubernetesNativeSessionClient use the same
removed Configuration accessors and will fail the same way on Flink 2.x.
Neither is reachable in the environment this was verified in, so they are
left for a change that can be tested.1 parent 9ddda84 commit 72fa842
12 files changed
Lines changed: 218 additions & 30 deletions
File tree
- streampark-common/src/main/java/org/apache/streampark/common/util
- streampark-console/streampark-console-service
- streampark-flink
- streampark-flink-client
- streampark-flink-client-api/src/main/java/org/apache/streampark/flink/client
- bean
- conf
- streampark-flink-client-core/src/main/java/org/apache/streampark/flink/client
- impl
- tool
- trait
- streampark-flink-packer/src/main/java/org/apache/streampark/flink/packer/pipeline
- streampark-flink-proxy/src/main/java/org/apache/streampark/flink/proxy
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
46 | | - | |
| 47 | + | |
47 | 48 | | |
48 | 49 | | |
49 | 50 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
372 | 372 | | |
373 | 373 | | |
374 | 374 | | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
375 | 381 | | |
376 | 382 | | |
377 | 383 | | |
| |||
Lines changed: 4 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
40 | | - | |
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| |||
263 | 263 | | |
264 | 264 | | |
265 | 265 | | |
266 | | - | |
| 266 | + | |
267 | 267 | | |
268 | 268 | | |
269 | 269 | | |
| |||
296 | 296 | | |
297 | 297 | | |
298 | 298 | | |
299 | | - | |
| 299 | + | |
| 300 | + | |
300 | 301 | | |
301 | 302 | | |
302 | 303 | | |
| |||
Lines changed: 57 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
Lines changed: 20 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
165 | 166 | | |
166 | 167 | | |
167 | 168 | | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
175 | 188 | | |
176 | 189 | | |
177 | 190 | | |
| |||
Lines changed: 3 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
30 | | - | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
161 | 161 | | |
162 | 162 | | |
163 | 163 | | |
164 | | - | |
| 164 | + | |
165 | 165 | | |
166 | | - | |
| 166 | + | |
167 | 167 | | |
168 | 168 | | |
169 | 169 | | |
| |||
Lines changed: 69 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
39 | 40 | | |
40 | 41 | | |
41 | 42 | | |
| 43 | + | |
42 | 44 | | |
43 | 45 | | |
44 | 46 | | |
| |||
69 | 71 | | |
70 | 72 | | |
71 | 73 | | |
72 | | - | |
73 | 74 | | |
74 | 75 | | |
75 | 76 | | |
| |||
221 | 222 | | |
222 | 223 | | |
223 | 224 | | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
224 | 246 | | |
225 | 247 | | |
226 | 248 | | |
| |||
235 | 257 | | |
236 | 258 | | |
237 | 259 | | |
238 | | - | |
| 260 | + | |
239 | 261 | | |
240 | 262 | | |
241 | 263 | | |
| |||
385 | 407 | | |
386 | 408 | | |
387 | 409 | | |
388 | | - | |
| 410 | + | |
389 | 411 | | |
390 | 412 | | |
391 | 413 | | |
| |||
417 | 439 | | |
418 | 440 | | |
419 | 441 | | |
420 | | - | |
| 442 | + | |
421 | 443 | | |
422 | | - | |
423 | | - | |
| 444 | + | |
| 445 | + | |
424 | 446 | | |
425 | 447 | | |
426 | 448 | | |
| |||
508 | 530 | | |
509 | 531 | | |
510 | 532 | | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
511 | 548 | | |
512 | 549 | | |
513 | 550 | | |
| |||
533 | 570 | | |
534 | 571 | | |
535 | 572 | | |
536 | | - | |
537 | | - | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
538 | 587 | | |
539 | 588 | | |
540 | 589 | | |
| |||
587 | 636 | | |
588 | 637 | | |
589 | 638 | | |
590 | | - | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
591 | 650 | | |
592 | 651 | | |
593 | 652 | | |
| |||
596 | 655 | | |
597 | 656 | | |
598 | 657 | | |
599 | | - | |
600 | | - | |
| 658 | + | |
601 | 659 | | |
602 | 660 | | |
603 | 661 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
47 | 48 | | |
| 49 | + | |
48 | 50 | | |
49 | 51 | | |
50 | 52 | | |
| |||
Lines changed: 3 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
55 | 56 | | |
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
| 60 | + | |
59 | 61 | | |
60 | 62 | | |
61 | 63 | | |
62 | 64 | | |
| 65 | + | |
63 | 66 | | |
64 | 67 | | |
65 | 68 | | |
| |||
0 commit comments