Error after upgrading from ACS 4.15 to ACS 4.20 #10207

KuasarCloud · 2025-01-19T16:02:31Z

KuasarCloud
Jan 19, 2025

When upgrading from 4.15 to 4.20 I cannot login anymore to the UI or interact with cloudmonkey

CMK

list domains
🙈 Error: (HTTP 530, error code 4250) Unable to find on DB, due to: Prepared statement needs to be re-prepared

UI Logs,

Nothing relevant, only

2025-01-19 10:52:04,093 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-4:[ctx-bae35c1b]) (logid:f2a9ebbd) Exception from : https://172.x.x.x:9090/clusterservice, method : null, exception : javax.net.ssl.SSLPeerUnverifiedException: Certificate for <172.x.x.x> doesn't match any of the subject alternative names: [fe80:0:0:0:5246:4aff:fe6e:8274, 172.x.x.x, co-x-phy-control-01.x.x, cloudstack.internal]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at com.cloud.cluster.ClusterServiceServletImpl.executePostMethod(ClusterServiceServletImpl.java:143)
	at com.cloud.cluster.ClusterServiceServletImpl.execute(ClusterServiceServletImpl.java:106)
	at com.cloud.cluster.ClusterManagerImpl.onSendingClusterPdu(ClusterManagerImpl.java:274)
	at com.cloud.cluster.ClusterManagerImpl$1.runInContext(ClusterManagerImpl.java:234)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Any idea? thanks

Ricardo Pertuz

KuasarCloud · 2025-01-20T15:05:46Z

KuasarCloud
Jan 20, 2025
Author

Guys, any help would be appreciated, I have a prod cluster down ;(
Thanks!

4 replies

DaanHoogland Jan 20, 2025
Collaborator

I do not know why this happens as the error message says that 172.x.x.x does not match any of fe80:0:0:0:5246:4aff:fe6e:8274, 172.x.x.x, co-x-phy-control-01.x.x, cloudstack.internal which it obviously does. I think your best bet is to renew the certificates on your MSses. and probably on your hosts.

However, the error seems to indicate an upgrade error and not an operational error. Where ther no error during the initial start of the MS?

KuasarCloud Jan 20, 2025
Author

thanks @DaanHoogland for replying, is there a procedure I can follow to renew it? all api call returns Unable to find on DB, due to: Prepared statement needs to be re-prepared

DaanHoogland Jan 20, 2025
Collaborator

there is a blog about it here: https://www.shapeblue.com/cloudstack-ca-framework/

but to be honest, I think this is not your problem.

Can you check what the contents of cloud .version is?

KuasarCloud Jan 20, 2025
Author

Here

MariaDB [cloud]> select * from version where id=45;
+----+----------+---------------------+----------+
| id | version | updated | step |
+----+----------+---------------------+----------+
| 45 | 4.20.0.0 | 2025-01-07 23:46:30 | Complete |
+----+----------+---------------------+----------+

KuasarCloud · 2025-01-21T17:11:28Z

KuasarCloud
Jan 21, 2025
Author

I have found this error in the DB, maybe a version issue with Mariadb? 10.5.11-MariaDB

DB Exception on: HikariProxyPreparedStatement@899781615 wrapping com.mysql.cj.jdbc.ServerPreparedStatement

2025-01-21 12:05:54,951 ERROR [c.c.a.q.d.UserAccountJoinDaoImpl] (qtp253011924-340:[ctx-8d971ff4, ctx-b306173f]) (logid:5f2a57ea) DB Exception on: HikariProxyPreparedStatement@899781615 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[196164]: SELECT user_view.id, user_view.uuid, user_view.username, user_view.password, user_view.firstname, user_view.lastname, user_view.email, user_view.state, user_view.api_key, user_view.secret_key, user_view.created, user_view.removed, user_view.timezone, user_view.registration_token, user_view.is_registered, user_view.incorrect_login_attempts, user_view.account_id, user_view.account_uuid, user_view.account_name, user_view.account_type, user_view.account_role_id, user_view.domain_id, user_view.domain_uuid, user_view.domain_name, user_view.domain_path, user_view.job_id, user_view.job_uuid, user_view.job_status, user_view.default, user_view.source, user_view.is_user_2fa_enabled FROM user_view WHERE user_view.account_id=2 AND user_view.account_type != 5  AND user_view.username LIKE x'61646d696e'  AND user_view.id != 1  AND user_view.removed IS NULL  ORDER BY user_view.id ASC  LIMIT 0, 500 java.sql.SQLException: Prepared statement needs to be re-prepared
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:130)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
	at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:972)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
	at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:434)
	at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:368)
	at com.cloud.utils.db.GenericDaoBase.searchAndCount(GenericDaoBase.java:1478)
	at com.cloud.utils.db.GenericDaoBase.searchAndCount(GenericDaoBase.java:1468)
	at jdk.internal.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at jdk.proxy3/jdk.proxy3.$Proxy395.searchAndCount(Unknown Source)
	at com.cloud.api.query.QueryManagerImpl.getUserListInternal(QueryManagerImpl.java:808)
	at com.cloud.api.query.QueryManagerImpl.searchForUsersInternal(QueryManagerImpl.java:724)
	at com.cloud.api.query.QueryManagerImpl.searchForUsers(QueryManagerImpl.java:659)
	at org.apache.cloudstack.api.command.admin.user.ListUsersCmd.execute(ListUsersCmd.java:90)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:831)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:652)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:354)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:157)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:154)
	at com.cloud.api.ApiServlet.doGet(ApiServlet.java:108)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:840)

0 replies

KuasarCloud · 2025-01-21T17:20:18Z

KuasarCloud
Jan 21, 2025
Author

I solved it by changing some variables in the mariadb config file

[mysqld]
table_definition_cache=4000
table_open_cache=8000

and in the shell

SET GLOBAL table_definition_cache = 4000;
SET GLOBAL table_open_cache = 8000;

My config was too low

table_definition_cache=400
table_open_cache=2000

0 replies

weizhouapache · 2025-01-21T18:22:41Z

weizhouapache
Jan 21, 2025
Collaborator

thanks @KuasarCloud for the update
closing

0 replies

Uh oh!

Error after upgrading from ACS 4.15 to ACS 4.20 #10207

Uh oh!

KuasarCloud Jan 19, 2025

Replies: 4 comments · 4 replies

Uh oh!

KuasarCloud Jan 20, 2025 Author

Uh oh!

DaanHoogland Jan 20, 2025 Collaborator

Uh oh!

KuasarCloud Jan 20, 2025 Author

Uh oh!

Uh oh!

DaanHoogland Jan 20, 2025 Collaborator

Uh oh!

KuasarCloud Jan 20, 2025 Author

Uh oh!

Uh oh!

KuasarCloud Jan 21, 2025 Author

Uh oh!

KuasarCloud Jan 21, 2025 Author

Uh oh!

weizhouapache Jan 21, 2025 Collaborator

KuasarCloud
Jan 19, 2025

Replies: 4 comments 4 replies

KuasarCloud
Jan 20, 2025
Author

DaanHoogland Jan 20, 2025
Collaborator

KuasarCloud Jan 20, 2025
Author

DaanHoogland Jan 20, 2025
Collaborator

KuasarCloud Jan 20, 2025
Author

KuasarCloud
Jan 21, 2025
Author

KuasarCloud
Jan 21, 2025
Author

weizhouapache
Jan 21, 2025
Collaborator