如何编写一个进程池bash shell

前端之家收集整理的这篇文章主要介绍了如何编写一个进程池bash shell前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。
我有超过10个任务执行,系统限制了最多可以同时运行4个任务。

我的任务可以像:
myprog任务名称

我如何编写一个bash shell脚本来运行这些任务。最重要的是,当一个任务完成时,脚本可以立即启动另一个任务,使运行的任务计数始终保持4。

我在考虑编写自己的流程池的同时,特别喜欢Brandon Horsley的解决方案,尽管我无法获取正确的信号,但我从Apache获得灵感,并决定尝试使用fifo作为前叉模型我的工作排队

以下功能是工作进程分叉时运行的功能

  1. # \brief the worker function that is called when we fork off worker processes
  2. # \param[in] id the worker ID
  3. # \param[in] job_queue the fifo to read jobs from
  4. # \param[in] result_log the temporary log file to write exit codes to
  5. function _job_pool_worker()
  6. {
  7. local id=$1
  8. local job_queue=$2
  9. local result_log=$3
  10. local line=
  11.  
  12. exec 7<> ${job_queue}
  13. while [[ "${line}" != "${job_pool_end_of_jobs}" && -e "${job_queue}" ]]; do
  14. # workers block on the exclusive lock to read the job queue
  15. flock --exclusive 7
  16. read line <${job_queue}
  17. flock --unlock 7
  18. # the worker should exit if it sees the end-of-job marker or run the
  19. # job otherwise and save its exit code to the result log.
  20. if [[ "${line}" == "${job_pool_end_of_jobs}" ]]; then
  21. # write it one more time for the next sibling so that everyone
  22. # will know we are exiting.
  23. echo "${line}" >&7
  24. else
  25. _job_pool_echo "### _job_pool_worker-${id}: ${line}"
  26. # run the job
  27. { ${line} ; }
  28. # now check the exit code and prepend "ERROR" to the result log entry
  29. # which we will use to count errors and then strip out later.
  30. local result=$?
  31. local status=
  32. if [[ "${result}" != "0" ]]; then
  33. status=ERROR
  34. fi
  35. # now write the error to the log,making sure multiple processes
  36. # don't trample over each other.
  37. exec 8<> ${result_log}
  38. flock --exclusive 8
  39. echo "${status}job_pool: exited ${result}: ${line}" >> ${result_log}
  40. flock --unlock 8
  41. exec 8>&-
  42. _job_pool_echo "### _job_pool_worker-${id}: exited ${result}: ${line}"
  43. fi
  44. done
  45. exec 7>&-
  46. }

你可以在Github get a copy of my solution。这是一个使用我的实现的示例程序。

  1. #!/bin/bash
  2.  
  3. . job_pool.sh
  4.  
  5. function foobar()
  6. {
  7. # do something
  8. true
  9. }
  10.  
  11. # initialize the job pool to allow 3 parallel jobs and echo commands
  12. job_pool_init 3 0
  13.  
  14. # run jobs
  15. job_pool_run sleep 1
  16. job_pool_run sleep 2
  17. job_pool_run sleep 3
  18. job_pool_run foobar
  19. job_pool_run foobar
  20. job_pool_run /bin/false
  21.  
  22. # wait until all jobs complete before continuing
  23. job_pool_wait
  24.  
  25. # more jobs
  26. job_pool_run /bin/false
  27. job_pool_run sleep 1
  28. job_pool_run sleep 2
  29. job_pool_run foobar
  30.  
  31. # don't forget to shut down the job pool
  32. job_pool_shutdown
  33.  
  34. # check the $job_pool_nerrors for the number of jobs that exited non-zero
  35. echo "job_pool_nerrors: ${job_pool_nerrors}"

希望这可以帮助!

猜你在找的Bash相关文章