Repository navigation
OutOfMemoryError when indexing AOSP #2074
Description
Hello.
I got OutOfMemoryError on a computer with 20GB RAM and Ubuntu 17.10.
07:15:27 SEVERE: 90417 successes (21.8%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
at org.opensolaris.opengrok.index.IndexDatabase.indexParallel(IndexDatabase.java:1092)
at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:474)
at org.opensolaris.opengrok.index.IndexDatabase$1.run(IndexDatabase.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
OS: Ubuntu 17.10.
Opengrok: 1.1-rc23
Java: OpenJDK 8
Tomcat: 8
BTW, is there any guide for indexing large projects like AOSP?
Activity
tarzanek commentedon Apr 19, 2018
I did index aosp in past with minor modifications, using JAVA_OPTS you can always up this
https://github.com/oracle/opengrok/blob/master/OpenGrok#L379
look at tuning guide:
https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases
I think what is in tuning guide is the lowest
I also had to setup some ignore patterns, alas I don't have them anymore and they are obsolete by now anyways ...
give it a shot using tuning guide, it will work
vladak commentedon Apr 19, 2018
vladak commentedon Apr 19, 2018
Maybe you can try to reduce the parallelism (via the
-T/--threadsoption passed toopengrok.jar) and see it that helps. Also, wondering if OpenJDK might behave differently than Oracle JDK.tarzanek commentedon Apr 20, 2018
well you're right Vlada,
the out of memory points out to a bug (be it in paralelism or leaking analyser)
AOSP is a good testing data set ...
idodeclare commentedon Apr 20, 2018
Well I’m not clear from the report if @kneep has raised the Java heap max from the default 2GB.
E.g. I have a 32GB test machine, but I cannot index the set of repos I care about without raising the heap to 10GB.
tarzanek commentedon Apr 20, 2018
well Idea starts at 756MB of ram and I was puzzled it could index big (java,scala) repos with ASTs, huge jar dependencies etc.
obbviously when I got it to 3GB it behaves much more snappier, but it did work with 756M too (although slow when I hit a certain spot)
so OpenGrok should be able to index with 2GB too (even with smaller) - lucene flushes after certain amounts of data(tunable), so that OOM really points to some leak (though I hoped we fixed all of analyzer leaks)
@kneep would you be able to figure out what files are open/indexed when the indexer crashes? (from logs?)
amckenna-pinterest commentedon Mar 14, 2019
I'm getting these with -Xmx12g on some repos.
amckenna-pinterest commentedon Mar 14, 2019
One of the repos has a 519M .json file, could that be related?
vladak commentedon Mar 14, 2019
Could well be. Try to put the file into ignored list and see what will happen.