Skip to content

GitLab

Switch branch/tag
This project is mirrored from https://*****@github.com/DataBiosphere/toil.git. Pull mirroring failed .
Last successful update . This branch has diverged from upstream.
  1. 23 May, 2022 1 commit
    • Adam Novak's avatar
      AGC integration fixes (#4110) · 303f240a
      Adam Novak authored 
      * Add a bunch of state tests

* Actually test S3/Celery, but just that combo

* Remove leading / from S3 state store paths

* Add a test to make sure we actually respect the base URL's key prefix

* Trim trailing whitespace

* Actually kill slow-to-stop workflows to fix #4111
      303f240a
  3. 20 May, 2022 3 commits
  5. 19 May, 2022 1 commit
  7. 18 May, 2022 1 commit
  9. 17 May, 2022 1 commit
  11. 11 May, 2022 1 commit
    • Adam Novak's avatar
      Allow persisting workflow state in WES server across container recreation (#4082) · c72a4bb9
      Adam Novak authored 
      * Implement an API for keeping track of workflow state metadata, including server node ID, on top of a swappable store

* Fix syntax

* Appease MyPy

* Have the backend be in charge of server identity

* Refactor tests so we can run them without a real Celery

* Allow specifying a state store to the server

* Implement an S3-based server StateStore

* Get S3 state storage working

* Cut down on state requests with a cache

* Fix syntax and typing

* Make the state machine much simpler and just time out CANCELING

* Fix allowing None out as a state

* Use python to find virtualenv so virtualenv can't fail to find python

* Fix typo

* Add a test for canceling state timeout recovery

* Add state store tests for file and S3

* Get client from resource properly

* Use lru_cache to memoize

* Handle file: URLs for state store

* Add a test for URL parsing

* Fix whitespace and restrict key characters

* Remove now-useless comment

* Backport to 3.7 functools

* Address review comments

* Use Aventer's Mesos repo now that it is a real repo

* Make sure you can't build without a TARGETARCH

* Ship distutils for Python3.7

* Try to work around broken package source

* Escape newlines

* Use the one Mesos deb that seems to work

* Remove nonfunctional apt option

* Go back to letting Mesos use its default file locations

The new debs for Mesos install the webui and libexec stuff properly where Mesos
can find them.

* Go back to using Aventer mesos that is built with a normal prefix
      c72a4bb9
  13. 20 Apr, 2022 1 commit
  15. 15 Apr, 2022 2 commits
  17. 08 Apr, 2022 1 commit
    • Jonathan Xu's avatar
      Add CWL tests under ARM (#4038) · c4dd55c3
      Jonathan Xu authored 
      

* Add CWL tests under ARM

* Add abstract class for cluster creation tests

* Add CWL on ARM tests to CI

* Inherit abstract cluster creation class in CWL tests on ARM

* Add needs_env_var decorator back

* Remove unnecessary methods from AbstractAWSAutoscaleTest

* Switch leaderNodeType to t4g.medium

* Remove spaces around TOIL_AWS_KEYNAME

* Install private key for CWL ARM tests

* Add clusterType arg to launchCluster

* Switch _require_env to  needs_env_var

* Create toil in tmp directory

* Increase leaderNodeType size
Co-authored-by: Adam Novak's avatarAdam Novak <anovak@soe.ucsc.edu>
      c4dd55c3
  19. 07 Apr, 2022 1 commit
    • Adam Novak's avatar
      Allow missing username and fix Docker build (#4077) · 7b513547
      Adam Novak authored 
      

* Test and implement a more robust user name finder

* Simplify testing all the lib stuff except Docker

* Try Ubutnu 20.04 and Mesos as suggested by the Mesos project

* Allow not having a python symlink to start

* Install dependencies that Mesos needs

* Stop installing OpenSSL libcurl because Mesos needs NSS libcurl

* Remove protobuf that just gets upgraded later

* Stop doing the Singularity dependency hacks since we now have the dependency available

* Tell Mesos where to find its web UI since the current builds can't.

* Tell Mesos agent where to find its libexec binaries

* Apply suggestions from code review
Co-authored-by: default avatarWilliam Gao <wlgao@ucsc.edu>
Co-authored-by: default avatarWilliam Gao <wlgao@ucsc.edu>
      7b513547
  21. 31 Mar, 2022 1 commit
  23. 30 Mar, 2022 1 commit
    • Adam Novak's avatar
      Reorganize Locking (#4059) · b3db63a3
      Adam Novak authored 
      * Create a coordination dir and use it for process names and cache databases

* Mount coordination dir and use it more

* Satisfy MyPy

* Do leader cleanup of single-machine deferred functions against coordination directory

* Bump Sphinx

@w-gao

 noted that a jinja2 v.s Sphinx 3 incompatibility is breaking all our builds, but that Sphinx 4 seems to work. In particular it's also breaking this PR, so I'm bumping it here.

* Resolve some documentation generation warnings

* Apply suggestions from code review
Co-authored-by: Lon Blauvelt's avatarLon Blauvelt <lblauvel@ucsc.edu>
Co-authored-by: Lon Blauvelt's avatarLon Blauvelt <lblauvel@ucsc.edu>
      b3db63a3
  25. 23 Mar, 2022 2 commits
  27. 17 Mar, 2022 1 commit
  29. 14 Mar, 2022 1 commit
  31. 10 Mar, 2022 1 commit
    • Adam Novak's avatar
      AGC Integration (#4039) · 97064368
      Adam Novak authored 
      * Implement untested AWSBatchBatchSystem

* Fix AWS Batch pylint errors

* Fix AWS Batch mypy errors

* Actually use our backoff to respond to SlowDown

* Fix API calling mistakes

* Rearrange session management so we can use the right primitives

* Let AWS Batch batch system have a region

* Refactor zone selection

* Make job names safe for Amazon Batch

* Properly find running and stopped batch jobs, and report status reasons

* Try to apply tags and set example scripts to parse args

* Get owner tagging working

Still to do:

* Ability to separate preemptible and non-preemptible jobs

* Ability to find the ephemeral storage on nodes that have it

* Ability to make sure jobs requiring a lot of storage run on queues with a CE
that provides a lot of storage, or maybe use an EFS filesystem

* Satisfy MyPy

* Fix some format strings missing fields

* Get AWSBatchBatchSystemTest::test_run_jobs to pass

* Give AWS Batch tests a longer time to start running jobs

* Actually use the subset arguments

* Add docs for the AWSBatchBatchSystem options

* Remove debug logging that isn't type-correct

* Harmonize specialized and general unit converters

* Get rid of decorator we don't actually decorate with

* Enstatic __ensafen_name

* Revise job-finding loop to break properly

* Replace double underscores with admonishments

* Fix spelling

* De-magic the API's min request limits

* Move boto-needing AWS imports to a new file

* Just use or instead of a fancy higher-order function

* Add logging to debug the busted Docker builder

* Fix conversion function variables

* Fix new MyPy error in threading

* Work around missing overloads in Boto3 stubs

* Find the new client and resource location

* Add a bunch of casts that MyPy insists on

* Fix an actual but MyPy caught where we used the wrong queries to remove policies

* Appease MyPy's fear of re-used loop variables

* Deal with roles appearing to be a single item

* Fit base-type-shaped peg into no-base-type-shaped hole

* Make Toil version be reported as a string in WES

* Test engine version presence and type

* Tolerate and require better-typed boto3 stubs

* Request workflows from S3 using libified job store utils

This will fix #4014.

* Add zip file test and make sure we have a broker before trying to use it

* Wait for workflows to succeed or fail and report logs in tests

* Tolerate and require better-typed boto3 stubs

* Require SDB and IAM stubs

* Adapt to establish_boto3_session moving

* Respect TOIL_OWNER_TAG in launch-cluster to fix #4019

* Remove unused import

* Use variable we already have

* Adjust copyright year

* Factor out job packing for contained executor

* Adjust KubernetesBatchSystem variable naming to match other contained batch systems

* Account for TOIL_OWNER_TAG in tag test

* Copy over utils from AGC WES adapter

* Teach Toil to parse AGC-style zip workflows

* Get zip workflow tests to pass and fix #4016

* Fix #4015 by letting the WES server be set to make job stores of other types

* Constrain stub submodule versions

* Add in missing imports

* Let the server tests be collected without the server extra

* Use typing_extensions to get TypedDict on 3.7

* Exit falsily if missing server dependencies

* Allow setting current region

* Fetch AWS AZ info from ECS metadata endpoint

* Import correct function

* Import module and check Fargate's vars

* Spell better

* Use string exit codes to fix #4030

* Switch to relative URIs to point to log files

* Log a bit on server startup

* Find retry_s3 in the tests

* Revert "Use string exit codes to fix #4030"

This reverts commit 50648cd3.

This will not work right with AGC until AGC fixes
https://github.com/aws/amazon-genomics-cli/issues/314 for full WES support.

* Fix asserting a member we removed wasn't falsey

* Find connection_reset

* Revert "Revert "Use string exit codes to fix #4030""

This reverts commit 23d8e9ce.

* Give containers more access to host for running more containers

* Fix typing

* Make default options templateable

* Revert "Make default options templateable"

This reverts commit ccea4b5a.

* Support --destBucket generation

* Use correct arguments name

* Go back to relative log URLs that support proxy path components

* Revert "Revert "Revert "Use string exit codes to fix #4030"""

This reverts commit 1621f7e9.

* Add 'toil' to generated file job stores

* Simplify zone and region determination

We will need to make the AGC startup script `export TOIL_AWS_REGION="${AWS_REGION}"`.
      97064368
  33. 02 Mar, 2022 1 commit
  35. 27 Feb, 2022 1 commit
  37. 24 Feb, 2022 2 commits
  39. 18 Feb, 2022 1 commit
  41. 16 Feb, 2022 1 commit
    • William Gao's avatar
      Update WES set up docs (#4027) · 4ac20410
      William Gao authored 
      * Update `docker-compose` example

* Update docker-compose.yml

* Update highlights in docs

* Remove unnecessary `--public_url` argument

* Include auth token in examples

* Review comments & update docs

* Fix mypy?

* Don't include relative paths

* Update docs

* Some review comments
      4ac20410
  43. 11 Feb, 2022 1 commit
  45. 09 Feb, 2022 1 commit
    • Kevin Hannon's avatar
      Issues/4005 add real time logs (#4031) · 5e9363df
      Kevin Hannon authored 
      * feat: add real time logging

* feat: WIP testing scatter

* feat: adding scatter bugfix for real time logs

* feat: cleanup of PR

* feat: format fix

* feat: format fix

* feat: black

* feat: WIP: unit testing

* feat: adding unit tests. WIP

* feat: remove logs

* feat: remove logs of cmd

* feat: added docs

* feat: test not quite working

* feat: unit tests finally working
      5e9363df
  47. 07 Feb, 2022 1 commit
  49. 04 Feb, 2022 1 commit
  51. 02 Feb, 2022 2 commits
  53. 01 Feb, 2022 1 commit
  55. 31 Jan, 2022 1 commit
  57. 29 Jan, 2022 1 commit
  59. 28 Jan, 2022 2 commits
    • Lukas Ho's avatar
      Fix assorted typos within assorted comments (#4023) · 5c431e5b
      Lukas Ho authored
      5c431e5b
    • Adam Novak's avatar
      AWS Batch Batch System (#3956) · 355d1529
      Adam Novak authored 
      * Implement untested AWSBatchBatchSystem

* Fix AWS Batch pylint errors

* Fix AWS Batch mypy errors

* Actually use our backoff to respond to SlowDown

* Fix API calling mistakes

* Rearrange session management so we can use the right primitives

* Let AWS Batch batch system have a region

* Refactor zone selection

* Make job names safe for Amazon Batch

* Properly find running and stopped batch jobs, and report status reasons

* Try to apply tags and set example scripts to parse args

* Get owner tagging working

Still to do:

* Ability to separate preemptible and non-preemptible jobs

* Ability to find the ephemeral storage on nodes that have it

* Ability to make sure jobs requiring a lot of storage run on queues with a CE
that provides a lot of storage, or maybe use an EFS filesystem

* Satisfy MyPy

* Fix some format strings missing fields

* Get AWSBatchBatchSystemTest::test_run_jobs to pass

* Give AWS Batch tests a longer time to start running jobs

* Actually use the subset arguments

* Add docs for the AWSBatchBatchSystem options

* Remove debug logging that isn't type-correct

* Harmonize specialized and general unit converters

* Get rid of decorator we don't actually decorate with

* Enstatic __ensafen_name

* Revise job-finding loop to break properly

* Replace double underscores with admonishments

* Fix spelling

* De-magic the API's min request limits

* Move boto-needing AWS imports to a new file

* Just use or instead of a fancy higher-order function

* Add logging to debug the busted Docker builder

* Fix conversion function variables

* Fix new MyPy error in threading

* Work around missing overloads in Boto3 stubs

* Find the new client and resource location

* Add a bunch of casts that MyPy insists on

* Fix an actual but MyPy caught where we used the wrong queries to remove policies

* Appease MyPy's fear of re-used loop variables

* Deal with roles appearing to be a single item

* Fit base-type-shaped peg into no-base-type-shaped hole

* Tolerate and require better-typed boto3 stubs

* Require SDB and IAM stubs

* Adapt to establish_boto3_session moving

* Remove unused import

* Use variable we already have

* Adjust copyright year

* Factor out job packing for contained executor

* Adjust KubernetesBatchSystem variable naming to match other contained batch systems
      355d1529
  61. 25 Jan, 2022 1 commit
  63. 24 Jan, 2022 1 commit
  65. 20 Jan, 2022 1 commit
    • Jonathan Xu's avatar
      Add launch cluster support for ARM (#3971) · e34118aa
      Jonathan Xu authored 
      * Add check for ARM in ami

* Add test for ARM ami search

* Add architecture to InstanceType

* Pass architecture to kubernetes setup

* Update EC2 Instance Types list

* Set default architecture in AWSProvisioner
      e34118aa