From the POEM@Home news feed:
POEM@HOME in the news
Tuesday, 9 June 2009 10:00 AM
On 14th of June 09 POEM@HOME will be featured in a short presentation on grid computing (http://en.wikipedia.org/wiki/Grid_computing) in the German TV program 3SAT Neues (http://blog.zdf.de/3sat.neues/) at 16:30 GMT+1. While the main focus of this program will be grid computing in general, the special efforts of the volunteer computing community will also be featured.
Message 3973 - Posted 27 Jan 2011 15:19:14 UTC - in response to Message 3972.
Hi,
1.) Are you happy with the development of POEM++?
Yes, very. In total POEM++ has proven to be very flexible. We can not only simulate Proteins, but also Proteins on surfaces, Membranes (for example for drug transfer, etc.) or even non-proteinogenic systems.
2.) Does the client deliver the expected results?
Yes, Energies of old - and P++ are compatible; POEM++ gives the same energies for the same proteins, but widened to the mentioned domain of 'not-proteins'.
3.) Is the performance alright? Somebody mentioned a 7fold increase.
The SASA (which was a roadblock in jpoem) is faster by far. The nice thing about the new SASA is near linear scaling. Jpoem had lots of problems, when it came to big proteins with runtime and especially memory. This was the reason why we couldn't calculate proteins bigger than 500+ amino acids on POEM@HOME. We can do that now.
4.) At the moment P++ is cpu only. Which cpu arch is the best fit for P++?
Well answering this is quite difficult. The linux client will benefit from everything up from SSE2, because the Matrix Vector operations are optimized for that. One thing is for sure however: Real Cores are better than hyperthreaded cores. In our future bigger clusters we would only put the big AMD CPUs at the moment, because there you simply get more CPUs per surface area.
5.) What are the problems in regards to P++ GPU development? Are the dev tools ok already?
The nice thing about GPU development is: You don't need special dev tools (or at least not so many). One of the reasons why the gpu development takes so much time is that some energy terms simply don't want to be parallelized. Even if of 6 energy terms 5 run 20 times faster, you won't realize a big difference if energy term 6 still takes as long as before. GPU cores cannot really talk with each other.
In regards to the app distribution there are some nice developments recently: From one hand ATI seems to bundle the OpenCL SDK with Catalyst nowadays and on the other hand some BOINC folks spend time in supporting OpenCL.
6.) When will the GPU app be released?
We can't say. At the moment we are preparing a simulation with a special forcefield, which will _only_ run on GPUs without parts on the CPU. It doesn't contain all interactions but the ones we want to investigate. I don't want to give a date, because in science some project comes in between all the time.
7)How big small is your group / the group of devs?
You find our group here:
http://www.research.kit.edu/biostruct/staff.php
At the moment we are quite big, but not everyone is working on the code all the time. Most of us execute simulations with P++. Lately got an additional person, who will now integrate the code for drug-screening.
8.) How good is the interaction with the BOINC people? How open is this interaction? CPU time means power and so I suspect that not everything is as open or as easy, correct? CPU time can be used for bad things afterall.
It is very open, there is also not a single feeling of competition because we all do different stuff. Lately I met some people from Rosetta@Home on a conference and we talked quite nicely about our and their algorithms. Among other things they also told me about their users who wanted to generate a movement against P@H, but thankfully their admins extinguished the flames.^^
We look at each others as scientists: If somebody cures cancer, it doesn't matter who he is; the only thing that matters is that the disease is cured.
Best,
Timo
Expected "hopefully" within the next 2 weeks:
Oct 28 , 2011 - New Client release, Server upgrade
We will release a new client by the end of next week. This client will be incompatible with the current client. Therefore we will starve the server of workunits the next week. Starting Monday no new workunits will be generated. We will then take the server offline on Friday to do a server upgrade to enable GPU client compatibility. This release will still be without GPU support. We expect the GPU support to be ready hopefully one week after. Sorry for the inconvenience.
Quote from: kashi on October 30, 2011, 06:26:02 PM
We expect the GPU support to be ready hopefully one week after.
Any news yet on which GPU's will be compatible?
Hopefully other medical based projects follow their lead. :thumbsup:
Hopefully it will be NVidia.... and from 250GTS, or 9800s...... cause poor ol' Gnatty can't handle the Beast that was in one of the other I7s.... 5780??... anyways, be nice to have something else to switch to, when I cross the 2.5 million mark with Einstein.
12 Feb 2012 | 10:35:02 UTC
Switching to fixed credits
As we discovered an overgranting for our resent workunits, we decided to change our credit system to fixed credits on Monday. All overestimated credit points will be normalized.
We apologise for the trouble caused.
The credits will be set as follows:
firstdrug: 80.8275865915366
gpucrystal: 2925.23047140266
rigiddock: 113.608330044946
barrelcompare: 20.6341535290809
I'm very glad POEM has stopped using the inconsistent, unfair and unreliable CreditNew system. The new POEM admins handled the change to fixed credits very quickly and professionally after the inevitable CreditNew disaster. Hopefully now WCG will reconsider their decision to use CreditNew with a GPU application before trouble strikes instead of after.
For anyone who tried the ATI/AMD application previously and gave up due to poor availability of work there has been no shortage of GPU work since the change to fixed credits.
POEM WU's have run dry.
There server status says they have "1" Tasks ready to send. ???
I mean I know i've been chopping into em', but I didn't think i'd crunched enough to run them dry? :jester:
@Jugnut
You broke their servers though. :furious:
biggrin I stopped crunching for a bit so there servers could catch up. So now there's enough work for all.. biggrin
:thumbsup:
POEM GPU Work shortages?.. Maybe for a long spell?
QUOTE: From: Thomas Koch. POEM ADMIN
------------------------------------------------------------------------------------------------------
Message 7389 - Posted: 26 Nov 2012 | 10:01:02 UTC
Hi everybody,
I have to apologize for the long period of silence regarding this topic, but I didn't want to give any misleading information before we discussed the problem here.
We are still waiting for our new server hardware, but unfortunately this won't fix the problem here entirely.
Fact is, these GPU calculations are running so rapid, that we have to evaluate real floods of information. We were talking about possibilities to send out more GPU work units, but at the moment there is no way to process it in time.
I want to thank all of you for spending so much computing time. It's absolutely okay to give your GPU power to another project now, there may be more use for it at the moment. The crystallization studies will still continue, but the follow-up projects with GPU support are not yet in a state to be sent out over the BOINC network, so the situation is not likely to get better in the near future.
I hope you can understand this, and are still willing to support us.
Best regards
Thomas
-------------------------------------------------------------------------------------------------------------
end quote: :cry2:
EDIT: CPU work still available ATM
There's still some GPU work but it's sporadic. But I can't snag any?
Anyone else with more info?
:thumbdown:
The current 'gpucrystal' tasks are drawing to an end with a new POEM GPU project planned for early next year. Hopefully they last out the AA but it sounds unlikely... :-\
boinc.fzk.de/poem/forum_thread.php?id=1028#9236 (http://boinc.fzk.de/poem/forum_thread.php?id=1028#9236)
New test POEM GPU work may start as early as next week. *fingers crossed*
QUOTE from Thomas Koch admin.. 23 Jan 2014
* Status Update *
Bad news first: The announced MOF simulation takes more time than expected.
We are currently working on a backup GPU project, which now seems to be more readily available.
The Test Server is up and running. Our IT Department is now configuring the Firewall for external access. I'll get the input files for our simulation next week, and will create POEM jobs as soon as possible. The URL will be announced when work units are prepared, because it only serves the dummy BOINC example_app at the moment.
If the jobs run fine on the test server, we will set them up on the live system the first week in February.
Thanks for your patience.
http://boinc.fzk.de/poem/forum_thread.php?id=1028&postid=9298#9298 (http://boinc.fzk.de/poem/forum_thread.php?id=1028&postid=9298#9298)
Well the new POEM site is up & running with all new GPU & CPU tasks. :thumbsup: Albeit just test jobs to make sure all is working well before full release.
But.. linux only ATM. :thumbdown:
This is the address for the test site. http://int-boinctest.int.kit.edu/poem/index.php (http://int-boinctest.int.kit.edu/poem/index.php) Windows work to come later this week.
Yes I've been following its development. Looks like it may be CPU bound again, but I kind of expected that.
For AMD/ATI after a quick look, one of the fastest is Dimitry's 7850/7870 Pitcairn and CPU time is almost equal to GPU processing time. Those who have probably starved their AMD/ATI GPU of CPU resources possibly by running CPU tasks at the same time or perhaps even using all CPU cores have a lower CPU time but a higher task time. This is assuming the test tasks were all the same length and not batches with different lengths.
Similar to the previous POEM GPU application this may mean recent model AMD/ATI lower end cards will be relatively more efficient and higher end cards will be hamstrung and need high CPU speed, high memory bandwidth and high PCIE bandwidth to try and minimise the throttling effect. Good for my 7790 Bonaire but a bit frustrating for those with higher end Tahiti and Hawaii class cards. However I'm not going to be GPU crunching in Linux so it remains to be seen if this will be similar when a Windows GPU application is developed.
I'm interested to see what the GPU load percentage is for a single task but nobody has mentioned it. Also not mentioned is if anyone has run multiple concurrent GPU tasks yet. If multiple concurrent tasks give more efficient performance then any throttling caused by CPU resource and bandwidth issues will be increased.
NVIDIA cards appear to have busy wait bug/feature as usual so you can't tell anything much about CPU usage from their figures.
The potential good news is that the GPU tasks are up to 100 or more times faster than the CPU tasks and as the tasks are the same it means GPU credit should again be generous assuming fixed credit is used. Although I'm not at all keen on the idea of CPU resource hog GPU applications, this is some recompense if you once again need to dedicate most or all of your CPU cores to the POEM GPU application to extract more efficient performance.
Regardless of any of the above the best news for me is that the work done is now for biology type research rather than for the development of OLED screens. The lack of biology/medical applications for AMD GPUs has had me doing Folding@Home for a while except for a recent little run on Einstein, so it will be good to have the option to also contribute to a favoured science type BOINC project on my 7790.;D
Yep sounds good kashi. If all things are equal I'll be getting stuck into POEM in a big way. :crazy
I wish I could try the linux GPU tasks, but that's not going to happen until a) I better learn Linux or b) if GPU crunching through a VM becomes a reality, until then I guess i'm out of luck.
Maybe in an ideal future you could say, crunch GPU grid in Linux, POGS on OSX, Collatz on Windows ect, all crunching on one PC in just one boinc window instance. A boinc hypervisor type deal maybe?
Possible? To much overhead perhaps?
In a way boinc is slowly moving that direction already.
Any thoughts?
Yes it would be good to split into different VMs according to which OS was most efficient on the projects you were currently crunching. I did it for a fair while crunching GPU projects on Windows BOINC and Linux efficient CPU projects on Linux BOINC in a VM.
You lose some efficiency using a VM but if the Linux application is much faster it more than compensates. However it slowed down some GPU projects unacceptably when I was crunching with 3 GPU cores. A single GPU core was fine though, although I never tried it on the previous POEM GPU application and it's likely that a very CPU demanding GPU app like that would possibly cause trouble. Of course if a GPU application is particularly CPU hungry you may be running few/no cores of CPU projects anyway even within the same OS.
As far as I know you can't use GPU project applications within a VM as the video drivers can't be emulated sufficiently, so that puts some restrictions on your flexibility as to which is your primary OS and which is your VM guest OS.
You would be able to swap things around I suppose with a dual boot setup and a VM within each OS, but I'm wary of stuff such as altering partitions and mucking around with boot managers and kyboshing the bootsector so that Windows no longer boots. Plus it's all time consuming and with only 1 computer any downtime means no work done.
I was keen to try running Hackintosh for POGS because the test OSX application Daniel was running appeared mighty fast. However he pointed out to me that it was a fast recent CPU without hyperthreading that was responsible for the majority of the speed up. I'd still like to give it a go one day if it's possible on my CPU, I just now looked at team member Paul's task times with Darwin 13.0.0 on his i5-3470S and it's smokin' fast for a 65W CPU. Yeh, yeh I know it's only 4 cores but even so it's very speedy per task. Sure to be messy, time consuming, possibly frustrating and perhaps even a bit dodgy, for any other project I wouldn't bother but POGS is Aussie and I've crunched millions of credit more of it than any other CPU only project. Therefore better to crunch it as fast as possible. Hmm, Snow Leopard, Lion, Mountain Lion, what sorcery is this? Anyway, looks like it's cool for cats.;D Oh no, iAtkos, MultiBeast, here comes trouble and I haven't even started yet.:pcwhack:
But you were mainly talking about the VM facility now integrated into BOINC for some projects and I haven't used that so can't comment much with any experience. It's certainly a great idea for long tasks without checkpoints like RNA World ones. I can't remember losing a long RNA World task but it was a horror not being able to reboot or shutdown for so long. Pretty quickly stopped me doing any more work for that project once the short tasks weren't available.
I did lose quite a few yoyo evolution tasks though, often near the end. Who says computers are not sentient and malevolent, the little electrons were often plotting my demise. I imagined them speaking in underpants gnomes' voices "Quick boys, that one's almost finished and there's no checkpoints, let's cause an exception error right now, that'll annoy him, haha.">:D
I updated VirtualBox to the latest version recently and installed PCLinuxOS LXDE as my trusty Dotsch UX had got too old and was missing too many recent libraries. Was disappointed in how it performed compared to Dotsch UX and haven't used it since, next time I'll try one of the Ubuntu variants. Dotsch UX on my CPU was always nicely speedy for BOINC compared to other Linux distros, wish Dotsch would release a new version but it's been a few years now, so seems unlikely.
I must say even after all these years of part time Linux usage, it remains a total pain when you lose familiarity with its foibles. Typing sudo every time to do just about anything more than basic configuration is just flaming idiotic. Even just manually updating to a recent version of BOINC that is not in the repositories yet is so much more complicated than it should be. Got something a bit different that's not easily fixed?, why "just compile your own kernel"; come on, get real Linux fu meisters, surely it would be more pleasant peregrinating in Tartarus.;D
Quote from: kashi on February 15, 2014, 05:38:59 PM
Regardless of any of the above the best news for me is that the work done is now for biology type research rather than for the development of OLED screens. The lack of biology/medical applications for AMD GPUs has had me doing Folding@Home for a while except for a recent little run on Einstein, so it will be good to have the option to also contribute to a favoured science type BOINC project on my 7790.;D
I was hoping this was the case, but I couldn't find anything regarding the new apps, where did you read it? :thumbsup:
Quote from: JugNut on February 15, 2014, 08:58:29 PM
I wish I could try the linux GPU tasks, but that's not going to happen until a) I better learn Linux or b) if GPU crunching through a VM becomes a reality, until then I guess i'm out of luck.
You could always try running it straight from a usb drive, I'm not sure which distro would be best, which ever one supports your card well I guess... ;D
@ kashi: Thanks for your input mate as always very much appreciated. I must admit to having to google iAtkos & MultiBeast, but glad I did. Interesting stuff indeed.
*LOL* I kinda skip read your post at first & for second I thought you said "electrons in underpants talking in hushed tones".. ??? That's one heck of a odd visual.. :rofl:
I think I may have wayy to much time on my hands? Although "speaking in underpants gnomes' voices" isn't a bad visual either.
Meanwhile back in what some call reality, if you ever find a suitable distro or a speedy way to change between OS's let me know.
@ Sean: Yea i've been meaning to try booting from a USB flash drive but that has most of the same problems as regular multi-booting that is the time you loose in crunching having to shut down & re-boot sort of negates much of the gains your trying to achieve in the first place.
If i'm to be totally honest the main reason I havn't tried many of these things is that i'd have to shut down crunching to try em' out, meaning lost crunching time.
Yea I know i'm a lost cause :faint:.
Quote from: Sean on February 16, 2014, 05:02:49 PM
Quote from: kashi on February 15, 2014, 05:38:59 PM
Regardless of any of the above the best news for me is that the work done is now for biology type research rather than for the development of OLED screens. The lack of biology/medical applications for AMD GPUs has had me doing Folding@Home for a while except for a recent little run on Einstein, so it will be good to have the option to also contribute to a favoured science type BOINC project on my 7790.;D
I was hoping this was the case, but I couldn't find anything regarding the new apps, where did you read it? :thumbsup:
It doesn't say it directly so you could be right, but I thought it could be reasonably deduced from what has been said in the News thread (http://boinc.fzk.de/poem/forum_thread.php?id=1028).
Firstly it says there has been a delay in the MOF simulation GPU app which is about finding carriers for a selective drug delivery in the human body.
Then it mentions the backup GPU project which is the one we're now discussing, currently being tested on the test server. This incorporates the SIMONA (SImulation of MOlecular and NAnoscale systems) code, their simulation framework as used in the existing POEM++ CPU app.
Further it is stated "I've just set up some of the known fastfolding jobs to test the compatibility." Fastfolding would relate to protein folding so the application is designed for this kind of work.
It's true the actual work units sent out are not necessarily going to be researching biological compounds or proteins, but there's no mention of any non biological related targets so far and the application includes SIMONA code as used for the existing poempp CPU application which is used for biological research. The term "gpucrystal" is not mentioned anywhere and as that research is finished now it is most probably not anything to do with OLED screens.
Finally, as a backup application for the delayed MOF simulation GPU app, it is likely to be related to their existing research which except for the previous gpucrystal GPU app has all been various types of biology research such as T3SS, peptides, hydrophobin, CASP, energystats, decoyrefold, decoyrelax, firstdrug, fold1em7, etc.
Perhaps my optimism about this is partly a projection from wishful thinking but generally the signs look good so far that it will be biology related. I probably should have been more cautious in that statement but the MOF app is coming and I know for sure about that one which made me think that this backup app is similarly biology related.
Wasn't the last round of POEM GPU work to do with biological nanotube research ???
Could be mistaken but I thought it was for OLED screen research, I remember there was some controversy when that was revealed, some were aghast and upset that it wasn't biology related like all other POEM applications.
@JugNut earlier post: Haha, yeh I also knew nothing about iAtkos & MultiBeast until earlier today, just thought I would mention them to show where the quest for POGS on OSX had led me so far (http://www.macbreaker.com/2013/01/iatkos-ml2-mountain-lion-virtualbox.html). Plus thought the name MultiBeast sounded excellent so thought I'd share it.
Have downloaded qbitorrent to get iAtkos but haven't installed it yet. Just checking it out a bit more first, never used a torrent program before, if I use one I always imagine heavily armed agents of the Murdoch/Brandis/MPAA alliance will be breaking down the door and bursting into my room the next day with "Stand up, hands on the desk, feet back and spread 'em. We've got you now, you despicable felon, we know you're about to upload copies of "SomeRubbishMovie" to the world." Plus Tooheys hackers will start using my computer as a StuxNet bot to disable Coopers brewing operations.;D
"Electrons in underpants", wow what a wild concept.:crazy
As to underpants gnomes, these hot nights with no sleep do sometimes get me raving on a bit more than usual.
Meanwhile, back in the "real" world, well at least relatively real as Buddhists sometimes say, I'll let you know if I ever manage to crunch some POGS on an OSX VM.
Thanks Kashi, I must have missed that, it does indeed look promising.
Also, this comment by Thomas is interesting:
Please note:
The Test System does not fulfill the security standards of our Live Server. The distributed files come with no warranty and are not properly signed, so attach on your own risk.
Credits will not be realistic and not have a meaning. This project should not be listed by BOINC Stats and similar websites.
Good to point that out to everyone Sean. :thumbsup:
Yes it's a bit different to some other BOINC test servers. Currently they only want testers to crunch a few tasks on GPU and CPU and if they complete and validate OK, then suspend the project. So far they are only testing if it works without error on different GPU models and CPU types. It is not a more exhaustive test with testers running lots of tasks like other projects' testing.
"...The only goal is to determine if the application fails on certain CPUs/GPUs.
So when you have successfully crunched two or three work units, please suspend the project. If any work units fail, please report it..."
The results so far for the available Linux applications look very positive: "Looking at the database it looks astonishing great at the moment :)"
GPU and CPU test applications are now available for Windows if anyone wishes to try a few before the official start of the AA.
from this post (http://boinc.fzk.de/poem/forum_thread.php?id=1028&postid=9382)
POEM test server to attach to is at: http://int-boinctest.int.kit.edu/poem
Most testers are reporting no problems and reasonable GPU load, although multiple GPUs setups may only work on one GPU.
I am full on POGS now so haven't tried it myself.
Had a mongrel time getting into the damn test evironment under my current login at POEM, so I recreated myself (I wish) and eventually crunched 7 WUs without problem. They validated but only took 48-50 seconds per work unit. These are obviously tests wus given the short duration but for 48 seconds 100 credits was awarded. Not valid credit because it is a test but may be an indication of what is to come. I didn't try multiple GPUs or multiple tasks per GPU because I have been running the mad cow project (MOO) and that does screw with GPU processing.
PS. Couldn't attach to BOINC@Australia on the TEST server
Perhaps the test server was having trouble when you were trying to log in. It is down currently as is POEM itself.
As far as I know the stats from the test are not going to be exported so if you have duplicated yourself and couldn't join the test team it doesn't matter.:jester:
I forgot the test GPU tasks run for under a minute on a recent GPU, I'll give a few a burl later tonight if the server comes back.
Edit: Aha, because it is a separate server, BOINC considers it a separate project so you need attach as a new user. I couldn't join the test team either because it doesn't exist.;D No need to create it I think as there is no stats export and the test only involves a few tasks each.
I did 5 tasks, they took about 35 seconds each. GPU load was erratic and varied for task duration, mainly low but did spike to 100% for a few seconds. Need longer running tasks to get a better idea but on my HD 7790 running at least 2 tasks concurrently looks likely to be necessary. Running only 1 did not cause the GPU core or GPU memory to rise to full speed from idle speed. In other words, average load for 1 task is low. CPU time of 11 seconds per task for 35 second tasks is moderately high, so throttling issues may occur unless sufficient CPU resources are allocated.
GPU WUs are available
Yea it's back but not quite like before. I left POEM open to receive GPU tasks & woke up late this morning with 10 units running at once from my old app_config. *LOL* They had been running over 3 1/2 hrs & had only gone 35% the way through.
It looks like if you run 2 or 3 at once on modern AMD cards they would take 2 - 2 .5hrs for 6,500 credit. Not sure about Nvidia's yet.
Not much being said yet at POEM other than dual Nvidia's aren't working yet & there seems to be other Nvidia problems as well.
So far for my rig its at most 2 or 3 at once. Not what I had expected but I'll test for a while & see what happens.
Crunch on..
All sorts of problems with the GPU side of POEM. SysOp doesn't know what is wrong and will have a look at it tomorrow. Most WUs seem to be failing at the end of processing.
Yep, I did 3 tasks and 2 errored. Near the end too, bah. :thumbdown:
On my 7790 single task gives erratic GPU load percentage and runs at reduced core clock of 1015 Mhz. Need to run 2 concurrently to get stable high GPU load and force default clock of 1075 MHz, but GPU memory still does not move off idle clock of 150 MHz.
CPU utilisation is much lower than previous application but still about 3% or 25% of a single HT CPU core. Could be better but common problem with OpenCL GPU applications. Still a big improvement as it means CPU tasks can be run at the same time as GPU tasks.
With 6 cores loaded with CPU tasks, 1 GPU task takes 2 hours and 2 tasks take 3 hours. Will try a single GPU task later today with no CPU cores allocated to other projects to see if it speeds up GPU processing at all. Edit: No I won't, no tasks left. ;D
With 2 concurrent tasks, credit rate has reduced by about 85% compared to previous application. :pcwhack:
Hi all,
FYI: I was surprised today to find a small amount of POEM GPU work on my PC's(6-10 WU's on each). Not from the test server either but from the proper POEM servers. Unfortunately I didn't notice them straight away & by the time I did & then tried to snag some more they had already run dry.
The bad news is they haven't changed much since last time, except GPU usage is now lower. On my HD 7970's, GPU usage is at about 52%. I then set up an app_config to run 2x at a time which seemed about right for a few tests.
Run time averaged about 2hrs 15mins when running 2x at a time. So half that for 1 WU of course.
1 x WU = 1hr 7mins
2 x WU's = 2hrs 15mins
All validated with no errors
Credit = 6,500 per WU
Unless there is a change credit will be poor to say the least.
Also I did try 3x WU's which did not go well at all, not only was there no gain from running 3 at a time, there was a big loss from doing so. More testing needs to be done on this when adequate work arrives.
So it appears POEM GPU WU's aren't far away... Good luck & crunch, crunch, crunch
New POEM GPU WU's are available for those wanting to give them a try. Come an get em' while they last..
Make sure you have BOINC configured so you have enough work to last over the outage.
Quote
Maintenance on 05/09/2015
The whole IT infrastructure of our institute will be maintained on Saturday, 5 September 2015, 6:00am to 4:00pm UTC. The POEM@HOME BOINC Server will not be reachable within this period.
Please make sure to have enough cached work units, if you want to keep your crunching machines fed.