Before I begin, let me say that I am in a rather niche position. I am developing a bioinformatics program that is designed for use on computers with 128GB of ram (here is a link the computers I am running https://pcpartpicker.com/list/9LzptJ). We have an entire stack of electron microscope scans that total to 26.5GB that we need to load into memory all at once. Clearly we have the memory to do this. I do not expect it to to take exactly 27 GB, but we should be able to load this stack given the memory we have. So im NOT an a laptop with 2GB ram trying to load this stack, we have enough physical memory to do this.
However, when ran in the IDE the program only gets to 96 GB (77% total system in use) before getting an out of memory error. Naturally I don’t want this behavior, and I need it to work the same for a huge stack as it does for a small stack. How can I increase the amount of memory given to processing? I have already changed the “increase maximum available memory to 128000MB”.
If you want to see the code, it is available here https://github.com/Jesse-McDonald/CASTER/tree/master and you can get a sample image stack to test this with here http://isoptera.lcsc.edu/~data/B1run02_png.zip or http://isoptera.lcsc.edu/~data/B1run02_png for individual images. Do note that the total size is 26.5 GB for all images so it will take a while to download, extract, and load.
Specifically, what error do you get?
Can you log the total / free / used memory?
here is the error
"java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at processing.core.PApplet.selectCallback(PApplet.java:6625)
at processing.core.PApplet.access$1(PApplet.java:6618)
at processing.core.PApplet$3.run(PApplet.java:6529)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: java.lang.OutOfMemoryError: Java heap space
at processing.core.PImage.(PImage.java:332)
at processing.core.PApplet.loadImage(PApplet.java:5467)
at processing.core.PApplet.loadImage(PApplet.java:5363)
at CASTER$EMStack.(CASTER.java:942)
at CASTER$EMStack.(CASTER.java:929)
at CASTER.load(CASTER.java:48)
… 21 more
"
I put the line output.println(Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()); in the loop that loads files, before the program crashed it created this log file
Summary
462572472
658577688
862846368
1260188928
1257895024
1692444984
1533835320
1968458280
2067586152
2502244640
2340018008
2773892816
2740711288
3176231008
3275544248
3472692336
3882161584
3882221728
4077264160
4151529712
4586056144
4681211544
5060895168
5090365224
5460196560
5425457416
5861473624
5895764568
6329122584
6764346232
6497852560
6868712064
6900382944
7271609864
7633849472
7500353904
7875870904
8309152072
7977051080
8412276704
8771104088
9129274296
9397897256
8983466144
9487341944
9919594672
10354594560
10002849984
10438047016
10193277808
10627535768
10878893216
11181363568
11432859064
11361686032
11682426648
11984978136
12219848712
12108869368
12411565096
12646367208
13076500296
13511460336
13133374400
13435726600
13670486040
13905611120
14140452640
14375716912
14216276928
14654127088
14825999432
15027655976
15457744248
15773403000
15531963008
15829716432
16259760112
16693005712
16985460264
16440196176
16906597104
17175701848
17445349552
17742447016
18176958488
18611495344
17840354944
18277334016
18471265976
18638253536
18976390520
19415751688
19847936128
20281746520
19583139240
19847465072
20273333136
20423445072
20640423048
21082600712
21225228592
21651136040
22081231952
21281136640
21706996312
22137073776
22569267632
22627071144
23044560080
23028829208
23463280912
23450216352
23867678264
23297475384
23715000816
24026480232
24465571512
24898806584
24852371208
25269812696
25312876848
25756253272
26186307832
25289569888
25733549216
26163594920
26140039080
26587352240
27013151544
27456869504
27203785400
27621180632
28047065560
27337751264
27755186656
28180993312
28611081120
29058980368
29484869608
29915001040
29469290920
29886783480
30312616168
30742649664
29658447272
30059043536
30494505896
30929934880
30572097208
31012095200
31446925376
31881771120
32316612496
31824936184
32250744280
32690168104
33120217480
33559737784
33989860192
34423102312
32786058128
33186802176
33616820864
34058089488
34483917440
34925265320
35355288088
35787532848
35080827056
35481424440
35911489312
36343553504
36776719744
37219233232
35612850040
36013539864
36439311824
36869327416
37317612672
37743510672
37017358576
37418017376
37843935296
38276091312
38709296440
38102615688
38510637240
38936524304
39366593792
39798684864
40249541248
39029280472
39436990216
39862931096
40293022512
40725187704
41175145200
40491023472
40899269024
41316801040
41746881784
42203774136
42621304328
43051391008
43508368032
43925863672
44355963976
44788138208
42470035752
42870683024
43288207304
43718307208
44150516592
44609477224
45026922424
45485894744
45903447792
46329269024
46759390224
47191698008
46023536576
46424158536
46841679176
47267477096
47697600872
48130880072
48582761456
49000139360
46503462448
46904095664
47321503512
47747327784
48178441120
48643094640
49060499384
49486299296
49916312408
48591517352
48958679712
49388587560
49828683456
50258738360
50698820224
51128740048
51560883768
52000959648
52431043688
50406218872
50773331344
51203199752
51644572816
52085939584
52511812400
52941707368
53383097752
53824422992
52372513872
52739517920
53165368336
53597393472
54043612536
54469300544
54915536640
55341240304
55771267376
54252835840
54630762568
55056482376
55501106712
55926809704
56371444560
56797301872
57229348576
57673979216
58099839336
56089242104
56456268552
56877759064
57309966016
57762288544
58179776064
58632104744
59049464544
59501942440
59919283920
58190436024
58568948360
58994809208
59424780280
59856824320
60307774208
60727685608
61178611576
61595946232
62046732376
62466780944
60592109008
60959164904
61376665064
61836931336
62263639840
62681007144
63141277088
63568002504
63985351672
64445661944
64872419592
65289848272
62716548624
63083775200
63501165328
63961092288
64387446936
64804771200
65264655648
65691017888
66108401408
66568311008
66994674928
67412033224
65199532408
65566563712
65983968464
66409743952
66872839984
67290220584
67715974024
68145905040
68608970696
69026334072
69452057792
69915105224
67768235520
68135256080
68570131968
69005000928
69439682088
69874559280
70309435648
70744317720
71179049568
71613955784
72048744592
72483634584
72918416000
73353364728
70304369280
70671412440
71071997368
71542524480
71943110728
72372889792
72838642856
73239106920
73671090448
74141782944
74542185288
74974218648
75444943344
75845546944
76277587808
73700502712
74067479072
74484768344
74914811920
75375635584
75792923360
76218615328
76650573408
77110341192
77527739352
77953369984
76158058872
76524977216
76942229776
77402759000
77820153664
78245770368
78706260912
79123471104
79549121568
80009600664
80427003592
77736262384
78103178000
78520426616
78980638376
79398072192
79823740040
80283953104
80701385696
81127074896
81587292072
82004554816
80047733032
80414629104
80831852760
81263908192
81719674120
82136895976
82566681472
83022586680
83439761576
83869502864
84325387152
82411642016
82778546024
83195804632
83656004264
84073267168
84498914872
84958929968
85376309392
85801992672
86262031480
86679270952
87105065664
84749639736
85150084072
85579888416
86011874800
86469053712
86886248192
87316184024
87774460936
88191622600
88656144768
89056628528
86995687472
87362493088
87779607856
88209436888
88667426088
89084657120
89514361928
89946183984
90404390944
90821552752
91251247056
91683192488
92141241920
92558691168
92988564664
93420672160
90080183000
90505960360
90971081448
91371703328
91833845744
92251092232
92676890200
93108983824
93571148648
93988543816
94414143200
92240158768
92607242112
93024507496
93454551784
93915125792
94332409544
93494636128
93861691312
94291771496
94076620088
94204003144
94384216608
ok, I just tried with a 13GB stack and got this crash
"
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at processing.core.PApplet.selectCallback(PApplet.java:6625)
at processing.core.PApplet.access$1(PApplet.java:6618)
at processing.core.PApplet$3.run(PApplet.java:6529)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: java.lang.OutOfMemoryError: Java heap space
at processing.core.PImage.init(PImage.java:266)
at processing.core.PImage.(PImage.java:234)
at CASTER$EMOverlay.(CASTER.java:832)
at CASTER$EMImage.(CASTER.java:612)
at CASTER.load(CASTER.java:48)
… 21 more
"
I figured out the problem, the stack of images I was using was stored using a gray scale pallet, in ARGB the image stack is 115GB, and PImage always internally uses ARGB regardless of what is required. I ended up having to implement my own compressed image type to import the images at a proper size.