This project is mirrored from https://*****@github.com/DataBiosphere/toil.git. Pull mirroring failed .
Last successful update . This branch has diverged from upstream.
  1. 23 May, 2022 1 commit
    • Adam Novak's avatar
      AGC integration fixes (#4110) · 303f240a
      Adam Novak authored
      * Add a bunch of state tests
      
      * Actually test S3/Celery, but just that combo
      
      * Remove leading / from S3 state store paths
      
      * Add a test to make sure we actually respect the base URL's key prefix
      
      * Trim trailing whitespace
      
      * Actually kill slow-to-stop workflows to fix #4111
      303f240a
  2. 20 May, 2022 3 commits
  3. 19 May, 2022 1 commit
  4. 18 May, 2022 1 commit
  5. 17 May, 2022 1 commit
  6. 11 May, 2022 1 commit
    • Adam Novak's avatar
      Allow persisting workflow state in WES server across container recreation (#4082) · c72a4bb9
      Adam Novak authored
      * Implement an API for keeping track of workflow state metadata, including server node ID, on top of a swappable store
      
      * Fix syntax
      
      * Appease MyPy
      
      * Have the backend be in charge of server identity
      
      * Refactor tests so we can run them without a real Celery
      
      * Allow specifying a state store to the server
      
      * Implement an S3-based server StateStore
      
      * Get S3 state storage working
      
      * Cut down on state requests with a cache
      
      * Fix syntax and typing
      
      * Make the state machine much simpler and just time out CANCELING
      
      * Fix allowing None out as a state
      
      * Use python to find virtualenv so virtualenv can't fail to find python
      
      * Fix typo
      
      * Add a test for canceling state timeout recovery
      
      * Add state store tests for file and S3
      
      * Get client from resource properly
      
      * Use lru_cache to memoize
      
      * Handle file: URLs for state store
      
      * Add a test for URL parsing
      
      * Fix whitespace and restrict key characters
      
      * Remove now-useless comment
      
      * Backport to 3.7 functools
      
      * Address review comments
      
      * Use Aventer's Mesos repo now that it is a real repo
      
      * Make sure you can't build without a TARGETARCH
      
      * Ship distutils for Python3.7
      
      * Try to work around broken package source
      
      * Escape newlines
      
      * Use the one Mesos deb that seems to work
      
      * Remove nonfunctional apt option
      
      * Go back to letting Mesos use its default file locations
      
      The new debs for Mesos install the webui and libexec stuff properly where Mesos
      can find them.
      
      * Go back to using Aventer mesos that is built with a normal prefix
      c72a4bb9
  7. 20 Apr, 2022 1 commit
  8. 15 Apr, 2022 2 commits
  9. 08 Apr, 2022 1 commit
    • Jonathan Xu's avatar
      Add CWL tests under ARM (#4038) · c4dd55c3
      Jonathan Xu authored
      
      
      * Add CWL tests under ARM
      
      * Add abstract class for cluster creation tests
      
      * Add CWL on ARM tests to CI
      
      * Inherit abstract cluster creation class in CWL tests on ARM
      
      * Add needs_env_var decorator back
      
      * Remove unnecessary methods from AbstractAWSAutoscaleTest
      
      * Switch leaderNodeType to t4g.medium
      
      * Remove spaces around TOIL_AWS_KEYNAME
      
      * Install private key for CWL ARM tests
      
      * Add clusterType arg to launchCluster
      
      * Switch _require_env to  needs_env_var
      
      * Create toil in tmp directory
      
      * Increase leaderNodeType size
      Co-authored-by: Adam Novak's avatarAdam Novak <anovak@soe.ucsc.edu>
      c4dd55c3
  10. 07 Apr, 2022 1 commit
    • Adam Novak's avatar
      Allow missing username and fix Docker build (#4077) · 7b513547
      Adam Novak authored
      
      
      * Test and implement a more robust user name finder
      
      * Simplify testing all the lib stuff except Docker
      
      * Try Ubutnu 20.04 and Mesos as suggested by the Mesos project
      
      * Allow not having a python symlink to start
      
      * Install dependencies that Mesos needs
      
      * Stop installing OpenSSL libcurl because Mesos needs NSS libcurl
      
      * Remove protobuf that just gets upgraded later
      
      * Stop doing the Singularity dependency hacks since we now have the dependency available
      
      * Tell Mesos where to find its web UI since the current builds can't.
      
      * Tell Mesos agent where to find its libexec binaries
      
      * Apply suggestions from code review
      Co-authored-by: default avatarWilliam Gao <wlgao@ucsc.edu>
      Co-authored-by: default avatarWilliam Gao <wlgao@ucsc.edu>
      7b513547
  11. 31 Mar, 2022 1 commit
  12. 30 Mar, 2022 1 commit
    • Adam Novak's avatar
      Reorganize Locking (#4059) · b3db63a3
      Adam Novak authored
      * Create a coordination dir and use it for process names and cache databases
      
      * Mount coordination dir and use it more
      
      * Satisfy MyPy
      
      * Do leader cleanup of single-machine deferred functions against coordination directory
      
      * Bump Sphinx
      
      @w-gao
      
       noted that a jinja2 v.s Sphinx 3 incompatibility is breaking all our builds, but that Sphinx 4 seems to work. In particular it's also breaking this PR, so I'm bumping it here.
      
      * Resolve some documentation generation warnings
      
      * Apply suggestions from code review
      Co-authored-by: Lon Blauvelt's avatarLon Blauvelt <lblauvel@ucsc.edu>
      Co-authored-by: Lon Blauvelt's avatarLon Blauvelt <lblauvel@ucsc.edu>
      b3db63a3
  13. 23 Mar, 2022 2 commits
  14. 17 Mar, 2022 1 commit
  15. 14 Mar, 2022 1 commit
  16. 10 Mar, 2022 1 commit
    • Adam Novak's avatar
      AGC Integration (#4039) · 97064368
      Adam Novak authored
      * Implement untested AWSBatchBatchSystem
      
      * Fix AWS Batch pylint errors
      
      * Fix AWS Batch mypy errors
      
      * Actually use our backoff to respond to SlowDown
      
      * Fix API calling mistakes
      
      * Rearrange session management so we can use the right primitives
      
      * Let AWS Batch batch system have a region
      
      * Refactor zone selection
      
      * Make job names safe for Amazon Batch
      
      * Properly find running and stopped batch jobs, and report status reasons
      
      * Try to apply tags and set example scripts to parse args
      
      * Get owner tagging working
      
      Still to do:
      
      * Ability to separate preemptible and non-preemptible jobs
      
      * Ability to find the ephemeral storage on nodes that have it
      
      * Ability to make sure jobs requiring a lot of storage run on queues with a CE
      that provides a lot of storage, or maybe use an EFS filesystem
      
      * Satisfy MyPy
      
      * Fix some format strings missing fields
      
      * Get AWSBatchBatchSystemTest::test_run_jobs to pass
      
      * Give AWS Batch tests a longer time to start running jobs
      
      * Actually use the subset arguments
      
      * Add docs for the AWSBatchBatchSystem options
      
      * Remove debug logging that isn't type-correct
      
      * Harmonize specialized and general unit converters
      
      * Get rid of decorator we don't actually decorate with
      
      * Enstatic __ensafen_name
      
      * Revise job-finding loop to break properly
      
      * Replace double underscores with admonishments
      
      * Fix spelling
      
      * De-magic the API's min request limits
      
      * Move boto-needing AWS imports to a new file
      
      * Just use or instead of a fancy higher-order function
      
      * Add logging to debug the busted Docker builder
      
      * Fix conversion function variables
      
      * Fix new MyPy error in threading
      
      * Work around missing overloads in Boto3 stubs
      
      * Find the new client and resource location
      
      * Add a bunch of casts that MyPy insists on
      
      * Fix an actual but MyPy caught where we used the wrong queries to remove policies
      
      * Appease MyPy's fear of re-used loop variables
      
      * Deal with roles appearing to be a single item
      
      * Fit base-type-shaped peg into no-base-type-shaped hole
      
      * Make Toil version be reported as a string in WES
      
      * Test engine version presence and type
      
      * Tolerate and require better-typed boto3 stubs
      
      * Request workflows from S3 using libified job store utils
      
      This will fix #4014.
      
      * Add zip file test and make sure we have a broker before trying to use it
      
      * Wait for workflows to succeed or fail and report logs in tests
      
      * Tolerate and require better-typed boto3 stubs
      
      * Require SDB and IAM stubs
      
      * Adapt to establish_boto3_session moving
      
      * Respect TOIL_OWNER_TAG in launch-cluster to fix #4019
      
      * Remove unused import
      
      * Use variable we already have
      
      * Adjust copyright year
      
      * Factor out job packing for contained executor
      
      * Adjust KubernetesBatchSystem variable naming to match other contained batch systems
      
      * Account for TOIL_OWNER_TAG in tag test
      
      * Copy over utils from AGC WES adapter
      
      * Teach Toil to parse AGC-style zip workflows
      
      * Get zip workflow tests to pass and fix #4016
      
      * Fix #4015 by letting the WES server be set to make job stores of other types
      
      * Constrain stub submodule versions
      
      * Add in missing imports
      
      * Let the server tests be collected without the server extra
      
      * Use typing_extensions to get TypedDict on 3.7
      
      * Exit falsily if missing server dependencies
      
      * Allow setting current region
      
      * Fetch AWS AZ info from ECS metadata endpoint
      
      * Import correct function
      
      * Import module and check Fargate's vars
      
      * Spell better
      
      * Use string exit codes to fix #4030
      
      * Switch to relative URIs to point to log files
      
      * Log a bit on server startup
      
      * Find retry_s3 in the tests
      
      * Revert "Use string exit codes to fix #4030"
      
      This reverts commit 50648cd3.
      
      This will not work right with AGC until AGC fixes
      https://github.com/aws/amazon-genomics-cli/issues/314 for full WES support.
      
      * Fix asserting a member we removed wasn't falsey
      
      * Find connection_reset
      
      * Revert "Revert "Use string exit codes to fix #4030""
      
      This reverts commit 23d8e9ce.
      
      * Give containers more access to host for running more containers
      
      * Fix typing
      
      * Make default options templateable
      
      * Revert "Make default options templateable"
      
      This reverts commit ccea4b5a.
      
      * Support --destBucket generation
      
      * Use correct arguments name
      
      * Go back to relative log URLs that support proxy path components
      
      * Revert "Revert "Revert "Use string exit codes to fix #4030"""
      
      This reverts commit 1621f7e9.
      
      * Add 'toil' to generated file job stores
      
      * Simplify zone and region determination
      
      We will need to make the AGC startup script `export TOIL_AWS_REGION="${AWS_REGION}"`.
      97064368
  17. 02 Mar, 2022 1 commit
  18. 27 Feb, 2022 1 commit
  19. 24 Feb, 2022 2 commits
  20. 18 Feb, 2022 1 commit
  21. 16 Feb, 2022 1 commit
    • William Gao's avatar
      Update WES set up docs (#4027) · 4ac20410
      William Gao authored
      * Update `docker-compose` example
      
      * Update docker-compose.yml
      
      * Update highlights in docs
      
      * Remove unnecessary `--public_url` argument
      
      * Include auth token in examples
      
      * Review comments & update docs
      
      * Fix mypy?
      
      * Don't include relative paths
      
      * Update docs
      
      * Some review comments
      4ac20410
  22. 11 Feb, 2022 1 commit
  23. 09 Feb, 2022 1 commit
    • Kevin Hannon's avatar
      Issues/4005 add real time logs (#4031) · 5e9363df
      Kevin Hannon authored
      * feat: add real time logging
      
      * feat: WIP testing scatter
      
      * feat: adding scatter bugfix for real time logs
      
      * feat: cleanup of PR
      
      * feat: format fix
      
      * feat: format fix
      
      * feat: black
      
      * feat: WIP: unit testing
      
      * feat: adding unit tests. WIP
      
      * feat: remove logs
      
      * feat: remove logs of cmd
      
      * feat: added docs
      
      * feat: test not quite working
      
      * feat: unit tests finally working
      5e9363df
  24. 07 Feb, 2022 1 commit
  25. 04 Feb, 2022 1 commit
  26. 02 Feb, 2022 2 commits
  27. 01 Feb, 2022 1 commit
  28. 31 Jan, 2022 1 commit
  29. 29 Jan, 2022 1 commit
  30. 28 Jan, 2022 2 commits
    • Lukas Ho's avatar
      5c431e5b
    • Adam Novak's avatar
      AWS Batch Batch System (#3956) · 355d1529
      Adam Novak authored
      * Implement untested AWSBatchBatchSystem
      
      * Fix AWS Batch pylint errors
      
      * Fix AWS Batch mypy errors
      
      * Actually use our backoff to respond to SlowDown
      
      * Fix API calling mistakes
      
      * Rearrange session management so we can use the right primitives
      
      * Let AWS Batch batch system have a region
      
      * Refactor zone selection
      
      * Make job names safe for Amazon Batch
      
      * Properly find running and stopped batch jobs, and report status reasons
      
      * Try to apply tags and set example scripts to parse args
      
      * Get owner tagging working
      
      Still to do:
      
      * Ability to separate preemptible and non-preemptible jobs
      
      * Ability to find the ephemeral storage on nodes that have it
      
      * Ability to make sure jobs requiring a lot of storage run on queues with a CE
      that provides a lot of storage, or maybe use an EFS filesystem
      
      * Satisfy MyPy
      
      * Fix some format strings missing fields
      
      * Get AWSBatchBatchSystemTest::test_run_jobs to pass
      
      * Give AWS Batch tests a longer time to start running jobs
      
      * Actually use the subset arguments
      
      * Add docs for the AWSBatchBatchSystem options
      
      * Remove debug logging that isn't type-correct
      
      * Harmonize specialized and general unit converters
      
      * Get rid of decorator we don't actually decorate with
      
      * Enstatic __ensafen_name
      
      * Revise job-finding loop to break properly
      
      * Replace double underscores with admonishments
      
      * Fix spelling
      
      * De-magic the API's min request limits
      
      * Move boto-needing AWS imports to a new file
      
      * Just use or instead of a fancy higher-order function
      
      * Add logging to debug the busted Docker builder
      
      * Fix conversion function variables
      
      * Fix new MyPy error in threading
      
      * Work around missing overloads in Boto3 stubs
      
      * Find the new client and resource location
      
      * Add a bunch of casts that MyPy insists on
      
      * Fix an actual but MyPy caught where we used the wrong queries to remove policies
      
      * Appease MyPy's fear of re-used loop variables
      
      * Deal with roles appearing to be a single item
      
      * Fit base-type-shaped peg into no-base-type-shaped hole
      
      * Tolerate and require better-typed boto3 stubs
      
      * Require SDB and IAM stubs
      
      * Adapt to establish_boto3_session moving
      
      * Remove unused import
      
      * Use variable we already have
      
      * Adjust copyright year
      
      * Factor out job packing for contained executor
      
      * Adjust KubernetesBatchSystem variable naming to match other contained batch systems
      355d1529
  31. 25 Jan, 2022 1 commit
  32. 24 Jan, 2022 1 commit
  33. 20 Jan, 2022 1 commit
    • Jonathan Xu's avatar
      Add launch cluster support for ARM (#3971) · e34118aa
      Jonathan Xu authored
      * Add check for ARM in ami
      
      * Add test for ARM ami search
      
      * Add architecture to InstanceType
      
      * Pass architecture to kubernetes setup
      
      * Update EC2 Instance Types list
      
      * Set default architecture in AWSProvisioner
      e34118aa