What is Sourcefinder?

Sourcefinder is about testing the performance and quality of the Duchamp Sourcefinding application. We've built a simulated cube of the sky containing various radio sources, and it's the job of Duchamp to work out where the sources are. We plan on running Duchamp over the whole cube of simulated data to work how many of the radio sources in the cube it can find. Duchamp will need to be able to identify correct sources while keeping false positives to a minimum.

As this project is in beta, we welcome any sort of feedback, advice, or bug reports. You can either make a post on the forums or send us an email at icrar.tsn.website@gmail.com . We're happy to hear from you.

Join Sourcefinder

User of the Day

User profile Profile Tuna Ertemalp
Born in Germany (Öhringen), raised in Turkey (Istanbul), all grown up in USA (Seattle). Boğaziçi University, Istanbul, Turkey: BS in Computer...

News

Update 26 April 2017
Added a field to the main user page that shows your total CPU time and average CPU time per result across all hosts.

Pushed out another set of work units for this week. I'm currently in the process of packaging up some additional work units, as we ran out fairly fast last week. The extra work units are currently transferring over to the main server, so I'll push them out later this week (probably tomorrow).

The stats for this first week are:

Total Cubes: 10416
Total Results: 19199
Total Canonical Results: 6737. 64.6793394777%
Average Results Per Cube: 1.84322196621

Good Results: 17607. 91.7079014532%
Bad Results: 1592. 8.2920985468%

Client Bad: 1562. 8.13584040835%
Client InProgress: 30. 0.156258138445%
Client Good: 17607. 91.7079014532%

Server Inactive: 0. 0.0%
Server Unsent: 3491. 18.1832387103%
Server InProgress: 371. 1.9323923121%
Server Over: 15337. 79.8843689776%
91% good results so far, which is fairly decent. It seems likely to me that the additional percentage of bad results are likely due to the increase in the number of work units that are sent out to each host. We've had a few people posting on the forums about getting too many work units. Hopefully the percentage of bad work units will settle down over the course of the next few weeks as people adjust their project preferences. If anyone else has had an issue with too many work units on any of their hosts, please see this thread.

Thank you, everyone.
26 Apr 2017, 7:16:07 UTC · Discuss

We're now ready for Production!
Yup, the day has come. Sourcefinder is finally moving from testing to production!

Most of the bugs and issues we've been having during testing have been ironed out to a level that I'm happy with.
All testing work units have been cancelled, and all work units and results have been archived.
We're now going to begin to crunch actual data, 681GB of it!

27G cubelets_1
27G cubelets_2
27G cubelets_3
27G cubelets_4
27G cubelets_5
27G cubelets_6
27G cubelets_7
27G cubelets_8
27G cubelets_9
27G cubelets_10
27G cubelets_11
27G cubelets_12
27G cubelets_13
27G cubelets_14
27G cubelets_15
27G cubelets_16
27G cubelets_17
27G cubelets_18
27G cubelets_19
27G cubelets_20
27G cubelets_21
27G cubelets_22
27G cubelets_23
27G cubelets_24
27G cubelets_25
8.6G cubelets_26
681G total

We're starting with cubelet sets 1 to 4.

A few other notes:

I've added a new process to the server to archive work unit and result data older than 2 weeks, so you wont be able to see any of that archived data on the website.

Any credits earned during testing will remain during production.

All production work units will be registered under run 10.

The limit on the number of work units sent to a single host will be increased. I'm planning on modifying the max work unit configuration from:
<max_wus_to_send>1</max_wus_to_send>
<max_wus_in_progress>2</max_wus_in_progress>
<max_ncpus>4</max_ncpus>
to
<max_wus_to_send>8</max_wus_to_send>
<max_wus_in_progress>3</max_wus_in_progress>
<max_ncpus>16</max_ncpus>

This should mean up to 16 CPUS on each machine will be used, and each user will be able to have up to 3 work units per CPU at any one time. This means the maximum number of WUs that a client can be running at any one time is now 48; just make sure you can crunch them all in time!
Additionally, each scheduler request cycle will provide up to 8 work units.
These numbers are tentative as well, if there are any issues or recommendations, I'm more than happy to change them.

A few future things:

I plan on finally updating theSkyNet website to accurately reflect the current state of the Sourcefinder project.

I'll be reporting the project stats each week, for those who are interested.

I'll definitely be pushing work units out every Wednesday, and sometimes on other days of the week if I get time.

My main focus from now on is fixing the issues with theSkyNet website, as it's been running on life support for miles too long.


Thank you everyone who has helped out with the Sourcefinder project so far, we greatly appreciate the work you've all done. It's because of all of you that this project was able to eventually reach production :).
19 Apr 2017, 1:22:01 UTC · Discuss

Update 12 April 2017
I haven't heard much back from anyone concerning our Virtual Box problems, so I've decided that we'll just have to put up with the odd hitch along the way.
Most of the errors were fairly uncommon, so hopefully we'll be able to deal with them as they come.

As of next week (19th April) I plan on moving Sourcefinder from beta to proper production. We'll still use this same server, so there'll be no need to re-attach or swap to another project.

Next week be erasing all of the work units we've run so far, and cleaning up the database in preparation for the production runs. I'm going not going to modify anyone's credit though, you all deserve to keep that for the work you've done so far :)

I'm going to let the current set of work units run for the rest of this week and compare the results to those from a local run I've been doing. I'm expecting identical or very close to identical results.

So thank you everyone for your efforts so far, and I look forward to finally getting this project complete :)
12 Apr 2017, 1:32:15 UTC · Discuss


Changelog 5 April 2017
Fixed a small bug with the assimilator.

It turns out that the project has been disabled the whole week, so the db dump hasn't been running and neither has the assimilator. I'm just waiting for everything to get up to speed again, then I'll push out some new work units and provide this week's stats.

I've been posting some of our virtualbox issues to the Boinc forums as well as the boinc_dev mailing list. I'm still waiting on responses.
5 Apr 2017, 6:17:49 UTC · Discuss


Changelog 29 March 2017
As per the recommendations made in this thread, I've made a few changes to hopefully prevent the file deleter from erasing the parameters files.

Our weekly stats are as follows:

Total Cubes: 6944
Total Results: 14575
Total Canonical Results: 6707. 96.5869815668%
Average Results Per Cube: 2.0989343318

Good Results: 13895. 95.3344768439%
Bad Results: 680. 4.66552315609%

Client Bad: 678. 4.65180102916%
Client InProgress: 2. 0.0137221269297%
Client Good: 13895. 95.3344768439%

Server Inactive: 0. 0.0%
Server Unsent: 0. 0.0%
Server InProgress: 239. 1.6397941681%
Server Over: 14336. 98.3602058319%


We've managed to hit the less than 5% error margin for work units, Congratulations!

I should also note that we have very few invalid work units. Over the course of week, the new validator only picked up 4 invalid work units, that's a pretty good sign.

I'm still in the process of running a number of work units locally so I can compare the BOINC results to the local results. I'll probably have enough local results by next week or the week after to start properly looking at the accuracy of the data coming back from BOINC.

I'm also looking for any advice people might have on where I can look for assistance regarding the various virtualbox issues we've been having. I tried the boinc_projects mailing list, although no one replied to my message. Next, I'm going to post on the main BOINC message boards, but if anyone knows a better place to ask for BOINC + virtualbox specific advice, I'm all ears.
29 Mar 2017, 0:37:14 UTC · Discuss

... more

News is available as an RSS feed   RSS


©2017 ICRAR