# Fermilab Frameworks Workshop (Live notes) :::info **Indico site**: https://indico.fnal.gov/event/59872/ ::: | Session block | Convener | Hand-watcher/Zoom recorder | Note taker | | :--------|:----:|:----:|:----:| | [Monday (before break)](#Monday-before-break)| Marc | Saba | Erica | | [Monday (after break)](#Monday-after-break) | Chris | Kyle | Marc | | [Tuesday (before break)](#Tuesday-before-break)| Kyle | Marc| Saba | | [Tuesday (after break)](#Tuesday-after-break) | Saba | Erica | Chris | | [Wednesday (before break)](#Wednesday-before-break) | Erica | Chris | Kyle | | [Wednesday (after break)](#Wednesday-after-break) | Kyle | Marc | Saba | ___ ## Monday (before break) #### art framework status (K. Knoepfel) - Kyle mentioned the migration to AL9 and Andrew Norman asked how legacy UPS products will be ported over. This will be discussed somewhat in Chris Green's talk and in the discussion on Wednesday. - Aaron asked about GPU support - will discuss on Wednesday. - Liz asked about documentation migration. Resource limitations was the short answer. She noted that there were issues going from Redmine to GitHiub and from TWiki to GitHub. Lynn noted her experience migrating LArSoft documentation to GitHub. She used a migration script. However, the best available was inadequate. Required a combination of fixing existing redmine pages and editing generated markdown pages. Wrapped the conversion script in our own handmade script to automate as much as possible. Required significant amount of work. art uses advanced redmine features that make it more difficult. #### CMSSW status and roadmap (C. Jones) - Clarification on CMSSW *Services* that, by policy, cannot change physics results (anything that does must come from `EventSetup` - the conditions system). Some examples... - Histogramming - Root i/o - CUDA configuration - Question from Adam Lyon: Given that GPUs are getting better and more common, would you have done things differently? Chris answered that MT and GPUs are complementary. #### Future framework directions (K. Knoepfel) * Giuseppe: asked about how modules that produce multiple data products, and perhaps apply selection would map to this. Short answer: we are trying to move away from that model. [requires more discussion...] * Andrew: do we need to re-org data into column-wise structures? * A: Not necessarily---the current Meld implementation is row-wise. But thinking in terms of data-product sequences opens up the possibility of column-wise structures and operations. * Dave Brown: likes this concept a lot. How does this translate to an end user doing analysis? They tend not to use frameworks at that point, and tend to work in row-wise manner. * A: do not see a large impact. Most analysis input is formed in framework job, and how that operates does not change. What this is about is removing impediments in producing that end-user input. * [Adam: Frameworks are generally targeted at Reconstruction jobs and DUNE Reco will need the features that Kyle describes] #### SciDAC-4 (G. Cerati) * Chris Green: have you tried with a modern version of gcc? * A: no. Needed to rewrite. In order to get GCC 11 to vectorize code, it had to be written in a way that was hard to understand and maintain. * Chris Jones: for internal alg parallelization, was there anything the framework could have done to make it easier? They had started using OpenMP but then had to switch to TBB to be compatible with art. Answer: was very easy to switch from OpenMP to TBB. * Marc Paterno: Did you try using.. * Kyle: mentioned complications getting TBB and argobots to cooperate. Is there anything that could have been done on the framework side to make this simpler? * Marc P: argobots was a technology to squeeze most out of the performance of the OS w/o talking to OS threads. Perhaps if the framework were so detached from the threading system, which could be put in as build-time option, then maybe this problem could have been avoided. It's not clear that the benefit would be worth the effort. * So experience gained which could be documented somehow. * This is noted in the papers that they will publish * Heidi Schellman: This is really cool, any chance to partially incorporate into LArSoft in short term so all expts can use together? ## Monday (after break) #### artdaq (K. Biery) * Kyle Knoepfel: what is the anticipated development of artdaq over the next few years? * Kurt: generally continue to support existing experiments, with a possibility of one new experiment coming on board. #### Mu2e Framework Requests (R. Kutschke) * Andrew Norman: interruptible module or interruptible trigger path? * Rob: interruptible module. * Chris Jones: clarify? * Rob: Want to be able to call "time" on an overlong module, segregate data for offline investigation. * Chris Jones (re deep copy data reduction): CMMSW can do this. * Rob: happy to steal. * Kyle: how are deep copies stored now? * Rob: new data products. Ptrs, etc. reseated manually. * Erica: is this really data reduction, or would Views work? * Rob: Definitely data reduction: need to throw out 99% of data. * Erica (re object extension): is this like giving data products "versions", or more like adding addressable blocks, for instance. * Rob: versions [but not clear if question was clearly stated..] So user could access either? Yes. Probably. * Chris Jones (re TimeTracker Wall Clock vs CPU): CMMSW can do this, but restrictions (within a single thread). * Rob: all of our algorithms are thread-hostile currently. * Andrew Norman: How much of an RTC do you need? * Marc Paterno, Rob: specifically CPU time: RT doesn't stop because CPU is doing something else; CPU time does. * Lynn Garren (re RNG): for specific extension: we need to consider effect on different types of RNG with different properties. * Kyle: looking to redo the system. * Marc Paterno (re C++ enums in Python analysis): how married are you to the SWIG implementation as opposed to what the Python looks like? * Rob: Ray (Culbertson)? * Ray Culbertson: not tied to SWIG. * Philippe Canal: did you try PyROOT to avoid an extra dependency? * Rob: PyROOT was not in use by the original requestor. * Marc Paterno (re sparse vector improvements): clarify? What do you need? * Rob: efficient cache access. * (several): `std::unordered_map`? STL may already have an answer for this. * Philippe (re avoiding autoparse): need to produce dictionary as C++ module. Some autoparsing may still be necessary with respect to "old" data products. * Rob, Philippe: clarification. * Vassil Vassilev <vvasilev@cern.ch> provided the following pointers and offered to consult (email or zoom): * https://github.com/root-project/root/blob/master/README/README.CXXMODULES.md * https://scholar.google.bg/citations?view_op=view_citation&hl=en&user=4poYWhEAAAAJ&citation_for_view=4poYWhEAAAAJ:UebtZRa9Y70C * https://scholar.google.bg/citations?view_op=view_citation&hl=en&user=4poYWhEAAAAJ&citation_for_view=4poYWhEAAAAJ:3fE2CSJIrl8C * https://scholar.google.bg/citations?view_op=view_citation&hl=en&user=4poYWhEAAAAJ&citation_for_view=4poYWhEAAAAJ:d1gkVwhDpl0C * Chris Green (re SQL DB size in `/var/tmp`): Not necessarily `/var/tmp`, just environment variable `TMPDIR` which defaults to `var/tmp`. All you need to do is define `TMPDIR` for your job. * Rob. We have a solution that works for us right now (`--timing-db`). Question if anything framework can do globally. * Marc Paterno (re RUCIO, etc.): what's your timescale? * Rob: in time for cosmics (July '24). ___ ## Tuesday (before break) #### NOvA (G. Davies) Gavin especially stressed the fact that NOvA will be collecting data through 2026, and because of effort issues finds that stability is important. Issues raised: 1. Inability to create associations between different objects of the same product type. 2. Documentation issues: a. exit codes are not well-explained b. the workbook is very out-of-date c. it is hard to find best-practices guidance * Chris Jones: Inter-experiment communication tools might be of great value. In CMS, we found users are happy to answer questions of other users. * Jake Calcutt (offline): Should an art slack/users' forum be set up? * Heidi Schellman: art slack would be awesome. but someone needs to pay for it. cost is ~10$/active user/month. # of active users might be low. Might want to share with FIFE as they also need user communication channels. Question about being able to 'reconfigure' a module on the fly to support event display. * Chris Jones: The CMS framework has such support specifically for your event display. 3. Updated "toy experiment" code could help address new-user documentaiton issues. 4. The ability to reload a job's FCL and reprocess events on the fly. This is desired explicity for the event display. 5. Streamlining access patterns when using associations. Adam: are people investigating use of things like GPUs, or are algorithms really stable? A: yes, but not many resources available, so didn't comment here. By and large, algorithms are stable. Andrew: though there is active development for trigger algs #### LArSoft (E. Snider) Erica notes that LArSoft (the collaboration) mostly does not have "strong opinions" about framework issues. This is because LArSoft (the shared software libraries) tries to separate algorithm code from framework code. Conceptually there is framework interface code and framework independent code, but in practice it is not strictly followed. LArSoft is not the dominant driver of the framework features. A few observations that were noted on the current framework are: difficult to manage memory for large events, asynchronous data/continuous readout stream of data, support of concurrency, and configuration tools. Current multithreading model seems to address concurrency needs. Chris Jones: Do you anticipate the need of making CPU working efficienctly with using GPUs (You are blocking the CPU, then you can't use that)? A: definitely an area to watch Tom Junk noted that the framework does not prevent the continuous data stream but have no support. Tom: analyzing continuous data in processable chunks involves chunking it, which often requires some kind of knowledge of features of the data being chunked. Not sure if general solutions are possible, but providing a way for users to write their own chunkers certainly has been very valuable. Metadata that describes batch jobs can it be handled by art (Heidi mentioned Kafka). CMS does support such a file specifically to feed back information to CMS' workflow management system. Kyle mentioned that this functionality will be provided Heidi: wouldn't it be great if all those cool things that Giuseppe talked about would be available in LArSoft? A: it's already there. Heidi suggested some education on using it would be useful. Chris Jones mentioned use of Python for configurations as used by CMS. #### cetmodules and Spack (C. Green) Dependency reduction (with use of cetmodules) simplifies Spack recipes. Chris covered the need of migrating from UPS to Spack. With Spack, we will make maximal use of the relocatable pre-built binaries. Several experiments have some adhoc builds using Spack (ICARUS, DUNE) Alma 9 support in progress. There are still a few hurdles that are being worked on. Liz commented on the use of pre-built recipes and issues involved in dealing with them. Heidi suggested if there can be good documentation and training for using Spack. Mu2e exercise at the end of June to make use of Spack (and not using cetmodules to build). David Brown: What does the distribution stream look like? A: On the issue of system libraries and the grid, DUNE is one of those experiments that shipped its own copies (using CVMFS instead of tarballs, but the same idea). Just recently however, we discontinued that because the containers used by default with our grid submission tools solved that problem for us. So we removed our dependencies on dune_oslibs and removed it from our setup script. Some interactive nodes however (my new desktop included) didn't have a complete set, so I spent some time installing the missing pieces. Not every user has the ability to install system packages on their interactive computers though. ## Tuesday (after break) #### SBN (S. Gardiner) * Lynn Garren: clarification: a good fraction of packages need to be rebuilt for GENIE changes, but not the totality. * Erica Snider: LArSoft has soemthing in its work plan to address this specific problem. * Lynn Garren: does HepMC3 adequately describe the kind of data produced by NuWro, etc. * Steve. study ongoing: we think so but want more eyes to spot omissions. * Chris Jones: integration issues: external process or integration into modules via library linking. * Steven G.: GENIE pretty well integrated, not sure about others. * CMS has generator modules which can run an external process and then read the data back into the framework process * Erica: so you're already seeing scaling issues with 4 TPCs? * Steven G.: in terms of performance, certainly. Not terrible, but not something to ignore. * Erica: so the increased data granularity discussed would be useful to you? * Steven G.: yes, definitely. * Kyle (re ML for reconstruction): difference between framework support and not getting in the way. Is there anything your people have in mind that the framework could do to make things easier? * Steven G.: upcoming slides. * Marc Paterno: is the training run in the art framework or is that external? * Steven G.: will need to ask. Currently inference algorithms do not change the nature of the art workflow. * Jake: do you know if you're using GPU as a service for any of the inference? * Steven G.: not entirely sure, but believe GPU use is separated from inference stage. * Erica: is there value in avoiding the offload (to HPC clusters)? * Steven G.: open for discussion. * Erica: moving stages to HPC doesn't necessarily mean taking them out of LArSoft. * Steven G.: work in progress, in flux. People are trying different things and getting work done; may need help to determine optimal path. * Chris Jones: what constitutes HPC-friendly? * Steven G.: able to leverage massive core count. * Chris J.: is there anything you think you might need from the framework? CMS already uses HPC sites as basically a big grid site and that has worked well. * Kyle: further discussion necessary; move on for now. * Lynn Garren: is POMS not FIFE? * Adam Lyon (nodding, clarified): yes, POMS is FIFE. * Steven G.: OK, LArSoft no, FIFE yes: error on slide. * Chris Jones: CMS has nanoAOD, does exactly what you describe. * Erica S. (to Steven G.): clarify? * Steven G.: would be nice to have data products know how to represent themselves in an Ntuple. * Philippe C.: Should look at ROOT's RNtuple rather than standard ROOT, third party or _ad hoc_. * Heidi Schellman: for Data Discovery, DUNE is thinking about a description for generators for use with sam/metacat. MINERvA had a simple version of this but getting a common way of describing GENIE version, parameters, ... for reproducibility and discovery would be great. Presumably this info is in the fcl files but probably needs to be extracted to be useful. #### DUNE ND (M. Muether) Currently largely framework free, but do want to move to an integrated framework, and are making progress in this direction. * For simulation, ND groups currentl using EDEPSIM, but are testing LArSoft as G4 wrapper. * Need common GENIE and G4 implementations for ND and FD to minimize confusion * ND LAr detector response simulation and ML reco are run on GPU for performance reasons. Flat scaling over broad range of track segments or hits. Always significantly better than CPU. * 5 reco chains under dev * ND-LAr * ND Lar Flow (calib) * MLReco (GPU/python) * SAND-RECO * C++ KLOE-based reco for ECAL * NDGAr * art/LArSoft based * TMS * Newest det, still under dev * Pandora * Working with LAr and SAND * Noted that all have varied timing reson, triggering, etc, so framework ust be capable of working with a variety fo data atom and timing structures * Lynn Garren: I assume you're aware that LArSoft has a PANDORA interface that is maintained by the PANDORA collaboration. * Mathew M.: currently PANDORA use is outside LArSoft. Will need to thread the needle on bringing everyone into the fold after rapid development phase. * Lynn Garren <interrupt>: need to note that NuSystematics uses the Nutools suite, does _not_ use LArSoft. * Mathew: understood, thanks for correction. * Lynn Garren: we have an email from Chris Marshall asking to present changes to NuSystematics. Does that relate to what you've been talking about? * Mathew: it folds in, we're participating in. Proposal is more to do with separating NuSystematics to make for easier multi-experiment distribution. * Kyle: would you be open into discussion on framework requirements from ND? * Mathew: development is beginning to stabilize, so will be open to discusion going forward. * Kyle: applicability to ND of previous requirements document? * Mathew: mostly, but things have moved on in last 18 months. #### DUNE FD (B. Viren) For early running, readout the entire module for localized trigger records---implies ~1 trigger record/file. Init stages of algs require data from only a single det unit, so 2 strategies * serial * parallel (multi-threading) * DAQ decimation (eliminating some DUs) leads to variability in job size over job lifetimes, so must allocate for worst case * SBND has demo'd this approach with LArSoft/WireCell TPC3 stage: merging of cluster across DUs * Framework must transition from per-DU loading to whole TPC loading * For extended TRs, may also need to merge over slices (eg, in time). Maybe DUNE production processing wants to use GPUs * will need GPU task queuing and scheduling * GPUaaS: eg w nVidia Triton Server * WirCell/ZeroMQ... * All have issues. To address these, believe an indep mini-framework is needed to ensure common interfaces and other requirements for task queuing and scheduling * Recommended that a cross-package tasking group could be available for registering tasks. * Kyle: what do you mean by "mini-framework?" Common scheduling system to avoid blocking? * Brett Viren: need have a shim—common interface that avoids lockin. * Pengfei: slide 7: in the parallel loading strategy, would adding a unit into the hierarchy be beneficial. * Brett: read in the entire localized TR (or slice). WireCell would go into a highly-threaded mode. Don't think we'd need lazy loading/hierarchy extension. * Pengfei: what about memory limitations? * Brett: 25x ProtoDUNE—about 100GiB of RAM, one whole node. Still problems. ___ ## Wednesday (before break) :::success The purpose of this discussion is to better understand the problems that need to be solved through framework evolution, and to clarify priorities as to what needs to be discussed and addressed. The following list is incomplete (ie, quasi-random), so may not include things that should be discussed. If this is the case, please bring it up at the meeting. In some cases, there were things presented where we believe the problem is understood or being discussed elsewhere, so no further discussion is needed here. This list is below. The remaining items (and implicitly any additional problems people would like to discuss) follows immediately thereafter. * The Mu2e wish list * Believe we understand the problems as stated. Plan to discuss details with Mu2e later. * The fcl extension request is heard. We feel this needs further discussion, but propose a later, dedicated forum. * DUNE ND * There are major integration questions here, among possibly others. There is an effort underway to try to better understand these and solve some of the top level integration problems, so propose deferring discussions to those efforts. * DUNE FD * There are many interesting points raised, but feel the issues have already been communicated at sufficient depth for our purposes here (which is not to say that further discussion is unnecessary). * Legacy UPS products after AL9 migration * Important question, but not in scope <br> Topics on the table <br> * Documentation (prioritize the art workbook, reference manual, wiki migrations, something else) * GPU support (Aaron H + others) * What is needed here that is currently not provided? * GPUaaS available. * Do we need non-blocking behavior? * Do we need to integrate ML training into framework workflows? * Do we understand the GPU workloads: total times + typical datasize in / out for instance (ie, is data transfer time important?) * May not be actionable unless we understand more of the specific problems * DUNE FD: see DUNE point above * Event reprocessing ability for an event display (G. Davies)? * There is history here in art (eg, reconfiguration). But there are hooks to replay an event. * Future framework development supporting AI/ML. * Need to understand what this means. * In case of ICARUS ML workflow, currently offloads from art / LArSoft for HPC steps. Is there value added in integration with art / LArSoft? * From ICARUS/SBND: making prod workflows HPC friendly would be "huge lift". Explore in detail what this means. ::: #### GPU discussion - Chris J: CMS is in the process of migrating to a thin wrapper that enables portability across devices for the same source code (Alpaka). Working on framework components that would make that simpler to deal with. Is this portability something that experiments want? - Rob K: Mu2e has no immediate needs for that but they would like it to be part of a toolkit should they want it. - Andrei G: Sounds great, but is skeptical due to Java experience. - Adam L: CCE has looked at this extensively. - Barnali C: What are the pros/cons of offloading LArSoft algorithms to GPU. - A: Potentially very large gains in efficiency. But there is complexity in coding and latency issues re. to offloading. - Jake C: It would be good to audit the algorithms that are GPU-izable. - Brett V: - Non-trivial to port from CPU to GPU (requires evaluation of what's worth it). - Depends on the site. If we have to keep the GPU busy for the given site, then you have to go to great lengths to use the GPUs well. - Andrew N: Bare CUDA happens at algorithms level. We don't understand interplay between how modules are scheduled to run on CPU vs. GPU. - Chris J: For HLT, they didn't have to change their scheduling. Modern GPUs do this pretty well. Working with NVidia about the threading issue (they use one big lock on the backend). CMS finds that NVidia scheduling is similar to TBB scheduling in terms of having enough tasks to run. - Amit B: HEP-CCE has done a survey on the experiments' effort to make their data model GPU friendly. Probably would be of interest to the group: https://github.com/hep-cce2/GPU-DM . If that link wont work, use this instead [https://github.com/physnerds/GPU-DM.git] - Tom J: Jobs tend to be more memory hungry than CPU hungry. Perhaps the GPU may need to be an avenue simply to gain more memory for the job. Can the framework be given a "hint" as to what GPU memory will be used a lot? - Giuseppe C: There are various GPU use cases (e.g.) algorithms to be used/invoked within the framework, and data to be offloaded for external ML framework processing. Which use cases must be supported? - Erica: The GPUaaS met the need for the _specific_ LArSoft use cases - Giuseppe C: It needs to be clear when a tool addresses a specific use case or when it is a generic solution. For generic solution adequate user support and advertisement are necessary - Brett V: Spectrum of uses cases for using GPUs (e.g.): using portability libraries, ML inference, bare CUDA usage in C++ code. - Krzysztof G: Modules using Geant4, hopefully in the near future, will be using GPUs (e.g., to track optical photons) in addition to their concurrent use of CPUs for other tasks. - Marc P: One of the primary uses of the framework is to be a scheduling engine. Are you amenable to having the framework decide dynamically whether a particular algorithm can run on a CPU or a GPU? Or do you require that the user can demand a particular algorithm runs on the CPU or the GPU, but not either? - _[Different viewpoints related to reproducibility.]_ - Aaron H (from Zoom chat): Beyond AI/ML simulation of optical photons in LAr detector is a great application to support GPU’s. Something like G4+Opticks. - Andrew N: Currently have troubles co-scheduling jobs with different external inputs (?). It would be a help for the framework to assist with that. - Chris J: Boundary between workflow management system and framework starts to get blurred. That's a separate discussion. - Heidi S: People should be thinking about a workshop that discusses GPUs at a broader level. - Ken H: Latest test on Perlmutter---inference server spun up by glidein (each node gets its own server). #### EMPHATIC (J. Paley) Requests mostly related to improving workflows * Documentation * Answers that avoid using jargon. Most of their collabs won't know art-isms, so phrasing things in a more generic way would be helpful * Make docs searchable * Update ToyExperiment and Workbook * It is updated to be compatible w current release. What do. you mean? * The code was hard to find. (It is not on GitHub along with the rest of art...) * Felt that there were things that were deprecated elsewhere, so led to conflicting advice. * Getting things to build is always difficult. Shedding light on cetbuildtools/cetmodules "magic" would be helpful. * Ability to dynamically reload a fhicl job configuration and reprocess event * Use case was event display, see result of changing reco params in real time * Noted that event displays from decades ago had this capability. Felt it was "one of the most useful and powerful aspects of framework" from that time. * Q: do you want to reload plugins? * A: no * Some base classes for an event display that experiments could use to quickly put together an event display. Eg, reloading event display, buttons that would advance to a specific event, etc. Things to make event displays "easy" for small experiments Barnali C: perhaps should restart art workshop to teach people how to use it. It is hard to learn how to do things, and with new things ## Wednesday (after break) #### Documentation discussion We have: 1. workbook 2. toy experiment code 3. a Redmine Wiki 4. Feature-specific documentation in headers Chris Jones: What is the reader's goal? Learning general new things, or task-based information? Learning how to do some task might be solved well by a general forum for all art-using experiments. CMS has something that does this. - General agreement that Slack does not meet this need. - Hypernews did this job in the past. - CSAID is working on a hypernews replacement that may help. - GitHub has some capabilities that may be useful. - Different people have different ways of learning and we should try to engage as many as possible. - The existing email list is still used and is archived and searchable. For any online forum, ability to identify best answers is critical Heidi Schellman presented slides with several points on documentation and communication (comments on documentation for computing in general). * Many people don't know what documenation exists; * Fermilab websites are no longer searchable from Google. * Documentation can go to: 1. readthedocs.io (some concern with whether this is permanent) 2. doxygen (good for C++) 3. github.io (sphinx or markdown) good for python Common step-by-step instructions/suggestions on how to do this consistently would be useful. Tutorials/recipes on use of the documentation systems is also important. A LXR server allowing code navigation is also widely popular. [CMS does proved LXR and Doxygen https://cmssdt.cern.ch/lxr/, https://cmsdoxygen.web.cern.ch/cmsdoxygen/] WireCell documentation is generated from emacs org-mode and served on github.io and also bnl.gov. DUNE DAQ is using a set of tools that invovles the CI system in verifying naming and formatting, and automatically builds the documents from markdown to HTML on readthedocs.io. The details of the implementation can be found in the [`DUNE-DAQ/docs` repository](https://github.com/DUNE-DAQ/docs). In short, this repository gathers all markdown files from monitored repos, and has a GitHub Action workflow which runs every 10 minutes to check if any markdown files have been changed, and pull in the updates. Readthedocs.io is configured to monitor the `DUNE-DAQ/docs` repo, and regenerate the pages accordingly. The generated documentation can be found [here](https://dune-daq-sw.readthedocs.io/en/latest/). Communication issues: we have many different communications mechanisms. * listservs * physical meetings * un-indexed web pages * misdirected ServiceNow tickets * github issues Heidi suggests making a Slack instance for IF-common software * art * LArSoft * Linux evolution * FIFE with dedicated channels for major software stacks and meeting announcements and channels with user support for those systems. Users would have to be on an IF experiment to join. Need a paid instance to keep history. Younger DUNE collaborators seem to favor this system. Chris Jones: CERN supports Mattermost rather than Slack. They also support a Discourse server rather than hypernews. Andrei G: relying on a commercial entity entails the risk that they might go out of business, and then that would lose the information. Heidi: Slack is good for rapid communication, but should not be treated as an archival system. Krzysztof G: Acknowledges that hypernews needs to be retired, but emphasizes the features like email interface without ads (which allows to have all the communication in email) and the sort which is chronological and not based on the popularity of the topic Tom J: Slack has too much direct message effect; questions directed to an individual do not help get the rest of the community educated. Heidi afterwards: Jake Calcutt points out that the root forum is really useful and we should find out how they do that. https://root-forum.cern.ch **Art workbook discussion** One issue has been that different experiments use different versions of art, and so need different versions of the workbook. Also, some experiments didn't distribute the toy experiment code; this meant they couldn't make use of much of the workbook. Much experimenter feedback has been we want "1 hour to first useful physics histogram". This means that the toy experiment work is not of use to that person. Mike Kirby: The UK groups have a very successful LArSoft school run ~2x per year. They have workbook they use. It is updated regularly because of the frequent presentations of the schoool. Perhaps they have some ideas that can be brought into the art workbook. This is a significant effort from the UK software group. CMS does a similar thing in the CMS data analysis school. The LPC coordinators at Fermilab put a great deal of effort into this. Lisa G: We also need to think about how we are training new people for scientific computing. Erica S: The UK LArSoft workshop/school is a very valuable resource. But it is not experiment-agnostic. Jacob C: A dedicated art-school in support of all experiments, but with breakout sessions specific to each experiment might be useful. The inverse, of an experiment-based school with a dedicated session on art with support from the art expert team could also be valuable. Tom J: We need to have both written materials and live schools. Krzysztof G: agrees with the need for constant update of the traning materials and documentation. Mentions the need to recognize people doing it and agrees that it is not free. Marc P: Everything related to tutorial and documentation is effort limited. Question for Adam: Is there something that the experiments ought to be doing to help the laboratory know that this is where they think effort should be applied? Adam: Experiments meet with Jim A, and this needs to be brought it to him. Having requests come from the spokesperson, and reach out to ALDs. Heidi: DUNE ops program setup in progress, and there will be an ask for what CMS does, that is provides a substantial amount of user support through their Ops program. Adam: Fermilab being the host lab for DUNE, here is an opportunity for us to do something about it. Barnali (from Chat): DUNE organizes such tutorial where we have lead teachers, for example, Heidi, Kirby, Tom, Steve, Ken and then there are mentors, mostly postdocs. If organizing an art workshop needs more manpower, postdocs from different experiments, who use art, can join as mentors. Rob K mentioned that they have a post Doc, from Caltech, who has a relationship with the San Diego people, and we have an Eve 7 based event display. and it does something very well but not everything. Giuseppe suggested that it might be worth looking at that ... Chris J mentioned that the eve7 code is not part of the framework code. it is a standalone piece of code. #### Wrap up We wanted to have discussion. So, at least from our perspective, with this workshop we had plenty of discussion. Expect to hear from us more and follow up on the specific topics. There are few upcoming meetings: * LArSoft coordination meeting: Tuesday June 13 (9am CDT) * HEP Software foundation meeting: June 28 (9am CDT) * 56 Annual Users meeting: June 28-30 * Next art stakeholders meeting: July 18