Proof
- New functionality
- Add support for processing
many datasets in one go in TProof::Process(const char
*dataset, ...).
Two options are provided:
- 'grand
dataset': the datasets are added up
and considered as a single dataset (syntax:
"dataset1|dataset2|...")
- 'keep
separated':
the datasets are processed one after the other; the user is notified in
the selector of the change of dataset so she/he has the opportunity to
separate the results. A new packetizer, TPacketizerMulti, has been
developed for this case: it basically contains a list of standard
packetizers (one for each dataset) and loops over them (syntax:
"dataset1,dataset2,..." or dataset1 dataset2 ...").
In
both cases, entry-list can be applied using the syntax
"dataset<<entrylist", e.g.
"dataset1<<el1|dataset2<<el2|".
The datasets to be processed can also be specified on one or multiple lines in a text file.
See http://root.cern.ch/drupal/content/working-data-sets
for more details.
- Add
support for automatic download of a package when available on the
master but not locally. The downloaded packages are store under <sandbox>/packages/downloaded
and automatically checked for updates against the master repository. If
a local version of the same package is created (using the
UploadPackage) the entry in downloaded is
cleared, so that the behaviour is unchanged.
The new functionality is described in http://root.cern.ch/drupal/content/working-packages-par-files
.
- Add
the possibility to remap the server for the files in a dataset. This
allows, for example, to reuse the dataset information for the same
files stored in a different cluster.
- Add a local cache for
TDataSetManagerFile. This is mainly used to improve the speed of
TDataSetManager::ShowDataSets, which is run very often by users and may
be very slow if the number of dataset is large. The cache is also used
to cache frequently received dataset objects.
- Add the possibility to audit the activity on the nodes via syslog. See http://root.cern.ch/drupal/content/enabling-query-monitoring
.
- New
packetizer TPacketizerFile generating packets which contain a single
file path to be used in processing single files. Used, for example, in
tasks generating files. The files are specified into a TMap - named
'PROOF_FilesToProcess' - containing the list of files to be generated
per host (the key is the host name, the value the TList of TObjString
with the files names). Workers are first assigned files belonging to
the list with host name matching the worker name. The map is
distributed to the master via the input list.
- Add support for
automatic setting of pointer data members to the relevant object in the
output list. The use of fOutputList->FindObject("name") in
TSelector::Terminate is not needed anymore for pointer data members,
e.g. histograms.
- Improvements
- Improve support for valgrind runs in PROOF-Lite
- Add the possibility to add files to a dataset. This is
achieved with a new option 'U' (for update) to RegisterDataSet.
- Add
methof TProof::GetStatistics to allow the client to retrieve the
correct values of fBytesRead, fRealTime and fCpuTime at any moment;
this will be used to setup a sort of ROOTmarks in stressProof .
- Several improvements in the test program 'stressProof'
and in the tutorials under 'tutorials/proof'
- Avoid
contacting the DNS when initializing TProofMgr as base class of
TProofMgrLite: it is not needed and it may introduce long startup
delays.
- Make TProof::LogViewer("") start the viewer for
a Lite session, in parallel to whats happen for TProof::Open("").
- Several
improvements in the handling of wild cards in the dataset manager; for
example, issuing a GetDataSet(...) on a dataset URI containign wild
cards will return a grand dataset sum of all the datasets matching the
URI.
- Add options to get a list of all dataset registered names
from ScanDataSets (option kList; the result is a TMap of {TObjString,
TObjString} with the second TObjString empty).
- Improved version of the PQ2 scripts; the scripts now invoke a dedicated ROOT application (named pq2) available under $ROOTSYS/bin .
- Add
support for recursive reading of group config files via the 'include
sub-file' directive. This allows to have a common part and, for
example, customize differently the quotas.
- Fix an issue with TTreeFriends. New tutorial showing how to use friends in PROOF.
- Package
management: add support for arguments in the SETUP function: it is
possible now to pass a string or a list of objects. The
TProof::EnablePackage interface has been extended to support this.
- Optimize
the validation step in the case not all the entries are required. The
validation step is stopped as soon as the requested number of events is
reached. If the parameter "PROOF_ValidateByFile" is set to 1, the
number of files is exactly what needed; otherwise the number of files
may exceed the number of files needed by (Number_Of_Workers - 1) .
- New directive 'xpd.datadir' to better control the user data directories and their permission settings.
- In TPacketizerUnit, add the possibility to exactly share the number of cycles between the workers. See the parameter PROOF_PacketizerFixedNum.
- Implement
a timer to terminate idle sessions. The timeout value is controlled by
the variable ProofServ.IdleTimeout (value in seconds). This variable
can be set for all sessions in the xproofd config file via the 'xpd.putrc' directive.
- Fixes
- Fix a bug in error status transmission which avoid
session freezing in some cases
- FIx
a few issues in libXrdProofd.so with handling of connection used for
admin operation: this should solve some cases where the daemon was not
responding.
- Fix a few memory leaks showing up when
running several queries in the same session
- Fix a few issues affecting the new sub-merging option
- Fix an issue preventing proper real-time notification
during VerifyDataSet
- Fix an issue with TQueryResult ordering (was causing
random 'stressProof' failures)
- Fix
an issue with TProof::AskStatistics (fBytesRead, fRealTime and fCpuTime
were not correctly filled on the client; the values on the master,
displayed by TProof::Print were correct).
- Fix several small issues affecting the handling of global
package directories
- Fix an issue with socket handling in the main event-loop
while sendign or receiving files via TProofMgr.
- Fix
a problem counting valid nodes in sequential or 'masteronly' mode,
generating the fake error message "GoParallel: attaching to candidate!"
- Fix a few issues with the length of Unix socket paths
affecting PROOF-Lite and xproofd on MacOsX
- Fix an issue with the release of file descriptors when
recovering sessions .
- Fix an issue with a fake error message ("Error in
<TROOT::cd>: No such file root:/") in PROOF-Lite when
issuing TProof::SetParallel().
- Fix a problem with negative values for 'workers still
sending' in PROOF-Lite .
- Fix locking issue while building packages locally.
- Fix issue setting permission and ownership of the dataset
user directories.
- Fix
a subtle bug affecting the (possibly rare) case when not all entries
are required and # entries does not correspond to an complete subset of
files (e.g. # entries = 1001000 with files of 100000 entries each). The effect was uncomplete processing (skipped events, magenta bar) or a session freeze.