Testing of BOINC version of theSkyNet

[Dataman]

The project will be out of work for a week (or so) while they check the results that have been returned.

[LawryB]

Just as well too 'cause I can feel the hot winds of California blowing on my back.   

[danhtruong]

Hi Dingo,

Just wondering, for your personal BOINCstats (theSkyNet POGS of 17,000 cobblestones or so) was that a direct conversion from theSkyNet java applet or did you with zero credit on theSkyNet POGS?

I'm still running theSkyNet from their website and just wondering if i should start running the BOINC theSkyNet instead (if its the same thing?) or continue to run from their website.

[Dingo]

I started from Zero on the BOINC project.  When the project is ported over to the University site from Amazon they say that the Badges and points from the non BOINC project should be coming over.

[LawryB]

I stand suitably repentant 

Read a couple of the posts on the POG forum last night about the number of WUs some people (called "suckers") were downloading.  I guess the purpose of an Alpha test is to get stuff back to the developer as soon as possible.  Pretty obvious now, but I didn't give it any thought when I downloaded heaps of WUs from the last batch.   Seems Kevin had to issue shorter WUs so he could get some returns.  I was feeling such a DH for not seeing the bleeding obvious that I aborted my 55 remaining WUs and sent Kevin a PM.
He was gracious enough to suggest that my actions were a good test for the BOINC system.   Amusing thing was that those 55 WUs lasted about 12 seconds in the download queue.

Just something to think about when the next batch get released perhaps.

@ Kashi.   Great response to the post regarding the time it takes to process work units.  Thank you.

http://ec2-23-23-126-96.compute-1.amazonaws.com/pogs/forum_thread.php?id=65#384 is well worth a look folks.

[tazzduke]

Greetings All

Dingo or other Senior Member could we maybe suggest to Kevin over at POGS of a limit of workunits per core like they do at LHC@Home and other projects, that way he can expect quick turnaround of workunits, this would also alleviate the sucker problem, I would go at say 2 or 4 per core, in my case a max of 16 workunits at one time. 

This maybe lifted once into either Beta or Production, by that time more crunchers have jumped onboard.  I find it disheartening when I come across a cruncher who has amassed over a 100 workunits and take anywhere up to a week to process.  In other words very long time in PV jail.

Any thoughts???

Regards
Tazzduke

Crunching for the benefit of Humantiy 

[Mike Mitchell]

I find it disheartening when I come across a cruncher who has amassed over a 100 workunits and take anywhere up to a week to process.  In other words very long time in PV jail.

My pet peeve are the people who download a hundred or so and then do nothing with those work units (in any project).  :furious: Like the idea Tazz

[kashi]

Re my post at the POGS forum. Most Intel computers running Windows will complete each step much faster than Sajjad Imam's AMD FX-8120. I have looked at one of his recently completed tasks and something seems awry. It took over 20 hours for a task that would complete in about 5 hours on my computer. I know Bulldozer architecture is inefficient on some applications due to the shared floating point core per module but that difference seems too high. His times of 35-40 minutes or longer per step are very slow and not consistent. This indicates severe contention losses and/or possible CPU overcommitment. Some people get their back up if I suggest using less BOINC cores in an attempt to reduce contention losses, so I rarely mention it now but I will make an exception in this case. If my suggestion is not appreciated at least I have tried to help.

If Kevin is keen to get the results back quicker then an option is to reduce the deadline from the present 7 days to 4 or 5 days. One problem with limiting the number of tasks per core is the variability of task lengths. If you get a batch of tasks with only a small number of steps (fits, pixels) then these complete very quickly on a fast computer. It means if the project has a temporary server outage or break in work availability then you are out of work. Same if your own internet connection is unreliable. I would not recommend a deadline of less than 3 days unless the project really needs it, because that can interfere with sharing with other projects due to tasks going into high priority mode. It may also cause problems for those with slower computers who download a batch of longer tasks. 

Restricting the number of tasks does address extreme cache hogs to some extent, but it still only partially solves the problem of new members who download a full batch of tasks and complete only a few or none at all and do not abort unfinished tasks but just let them all time out. These anti-abortion hogs cause a longer delay than those with a large cache but a turnaround time of 2-3 days. With a restriction of 4 tasks per core, someone with an 8 core computer who downloads a batch of tasks then only completes 2, ties up 30 tasks for the full 7 days and then the potential 7 day cycle starts again when the tasks time out and are reissued. When batches are large,  sometimes it can take longer than 7 days for a task to be reissued because it can remain in unsent state for some time.

Not everyone reads a project forum and realises that a project wishes a quick turnaround time. If the deadline is 7 days then that's how long some will take to finish tasks.

When a batch of work needs to be checked before more work is released then a shorter deadline is a good way to reduce the total turnaround time of first issue and timeout reissue tasks.

If the project has an astronomical amount of data to process then attracting and keeping a decent size number of contributors is vital. I understand that the project is only new and batches need to be checked at this early stage so interruptions to work availability are currently unavoidable. However now that the project is open to all it is important that work is reliably and consistently available as soon as possible. Lack of work causes contributors to switch to other projects and if "no work available" is the introduction to a project any new contributors receive then some of those may not return. Lack of consistent work availability may also disqualify a project from being considered by teams as their special project for a time (like our Aussie Assaults).

[tazzduke]

Greetings Kashi

Thankyou for your input, has given me a new understanding.

Regards
Tazzduke

[LawryB]

@Kashi

Thanks Kashi.

I noted your earlier post (to dataman) about reducing the number of cores crunching, tried it, and yes it was much more efficient and also had a positive effect on my GPU computing.  
There is a post on Poem (I think) about the efficiency of GPU/CPU use (I will try to find it again and post a link here).

I think I would prefer a reduced deadline and/or smaller WUs (for the testing period) rather than a limit in downloads though.

Early days yet though and Kevin seems to be heading in the right direction.

Mate, for every one person who gets upset about suggestions there are 10 more who read AND digest.  Please keep it up.

Cheers

Lawry

EDIT:  http://boinc.fzk.de/poem/forum_thread.php?id=706  The Poem link.

[kashi]

Ah yes, but POEM GPU is an extreme case. It needs more CPU than any other released GPU project. So much so in fact that if you wanted to do the maximum amount of work and were not concerned with also supporting some CPU projects, then the greatest efficiency is achieved by not running any CPU projects at all! This is rather revolutionary for BOINC in a sense as previously GPU projects were considered as an addition to CPU projects on multi core computers not as a stand-alone crunching concept.

As you are aware there are some GPU applications where CPU usage is very small, mainly the ATI/AMD ones based on CAL/Stream originally optimised with IL code by Andreas Przystawik such as Collatz and MilkyWay.

Even though one of the POEM GPU developers states: "our application can not run on GPUs only, but also needs CPU calculations between GPU steps. This is a major difference to other GPU projects hosted by BOINC, and causes the non-optimal usage of GPU and CPU by bandwidth limitations", I still believe the POEM application was developed on older previous generation GPU hardware and could be improved to better utilise recent more powerful GPUs by modifying a combination of sleep time, chunk size, kernel calls, etc. so as to reduce CPU throttling thereby increasing GPU load percentage and increasing efficiency. User adjustable parameters to enable fine tuning the GPU application to suit different hardware and user needs would be great but unfortunately are very unlikely to be provided. It appears as though it took them so long to develop the application that they are now reluctant or have insufficient resources to even attempt improving or refining it.

[JugNut]

My god Kashi every time I think i've made one step forward about understanding parts of Boinc I read one of your posts & realize i've not scratched the bloody surface.

I despise butt kissers but credit where credits due my friend "you rock"

Maybe you could suggest a "Boinc for dummies book" for the thick as a brick type such as me.

[Dingo]

Kashi -  Thank you for the explanations for this topic and for the many you have given in the past.  You are a valuable member of the team and able to explain things much better than myself.  I may know what you are saying but cannot say it in such an elegant and straight forward way as you do.  +1 for your participation in the Team and the forum.

[kashi]

Aw shucks, thank you both. :blush1:

There are many who know a lot about BOINC but do not have either the time or the inclination to explain it in detail. I have taken a great interest in the development of GPU applications and have been privileged to be able to help some of the developers by testing and reporting on development versions of GPU application versions. The detailed workings of the BOINC program code itself still baffle me often however even after all this time. biggrin

I have not been able to spend much time on the forum in the last few months due to my mother's illness. She has been in various hospitals for the last ten weeks since an operation to repair a massive hiatus hernia in June. Will be off to visit her again in an hour at Berkeley Vale which is about 30-40 minutes north of here. She has improved greatly in the last 2 weeks and should be coming home in about 10 days.

Had a mini drama here yesterday, when I got up I could hear the water running strongly and thought my brother was in the shower. So I'm eating my breakfast thinking, "the bastard has been in there a long time, he'll use all the hot water again." Meanwhile he was out on the back verandah playing with next door's cat and had been thinking the same thing about me. Turns out we had a burst water pipe connection along the side of the house. The plumber was cursing when he discovered someone had used a section of inferior grade white plastic pipe, instead of some better quality black plastic or copper pipe. He is based a long way away from here and did not have the special connector needed in his truck so went off to a local supply company to get one. They did not have one as they are rare now, so he had to put in a new section of copper pipe. We were thinking it was going to cost a fortune since it all took about 3 hours but he only charged $200. So all's well that ends well.  

[LawryB]

Great news your Mum has improved and will be home soon Kashi.