Submit
Path:
~
/
/
proc
/
thread-self
/
root
/
opt
/
alt
/
python35
/
lib
/
python3.5
/
site-packages
/
joblib
/
__pycache__
/
File Content:
_parallel_backends.cpython-35.pyc
,|�W�7 � @ s� d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z m Z d d l m Z d d l m Z m Z d d l m Z d d l m Z e d k r� d d l m Z d d l m Z Gd d � d e e � � Z Gd d � d e � Z Gd d � d e � Z Gd d � d e � Z Gd d � d e e � Z Gd d � d e e e � Z Gd d � d e � Z Gd d � d e � Z Gd d � d e � Z d S) z, Backends for embarrassingly parallel code. � N)�ABCMeta�abstractmethod� )� format_exc)�WorkerInterrupt�TransportableException)�mp)�with_metaclass)� MemmapingPool)� ThreadPoolc @ s� e Z d Z d Z e d d � � Z e d d d � � Z d d d d � Z d d � Z d d � Z d d � Z d d � Z d d d � Z d S)�ParallelBackendBasezEHelper abc which defines all methods a ParallelBackend must implementc C s d S)a� Determine the number of jobs that can actually run in parallel n_jobs is the number of workers requested by the callers. Passing n_jobs=-1 means requesting all available workers for instance matching the number of CPU cores on the worker host(s). This method should return a guesstimate of the number of workers that can actually perform work concurrently. The primary use case is to make it possible for the caller to know in how many chunks to slice the work. In general working on larger data chunks is more efficient (less scheduling overhead and better use of CPU cache prefetching heuristics) as long as all the workers have enough work to do. N� )�self�n_jobsr r �/_parallel_backends.py�effective_n_jobs s z$ParallelBackendBase.effective_n_jobsNc C s d S)zSchedule a func to be runNr )r �func�callbackr r r � apply_async* s z ParallelBackendBase.apply_asyncr c K s | | _ | j | � S)z�Reconfigure the backend and return the number of workers. This makes it possible to reuse an existing backend instance for successive independent calls to Parallel with different parameters. )�parallelr )r r r � backend_argsr r r � configure. s z ParallelBackendBase.configurec C s d S)z#Shutdown the process or thread poolNr )r r r r � terminate7 s z ParallelBackendBase.terminatec C s d S)z Determine the optimal batch sizer r )r r r r �compute_batch_size: s z&ParallelBackendBase.compute_batch_sizec C s d S)z1Callback indicate how long it took to run a batchNr )r � batch_size�durationr r r �batch_completed> s z#ParallelBackendBase.batch_completedc C s g S)z'List of exception types to be captured.r )r r r r �get_exceptionsA s z"ParallelBackendBase.get_exceptionsTc C s d S)a} Abort any running tasks This is called when an exception has been raised when executing a tasks and all the remaining tasks will be ignored and can therefore be aborted to spare computation resources. If ensure_ready is True, the backend should be left in an operating state as future tasks might be re-submitted via that same backend instance. If ensure_ready is False, the implementer of this method can decide to leave the backend in a closed / terminated state as no new task are expected to be submitted to this backend. Setting ensure_ready to False is an optimization that can be leveraged when aborting tasks via killing processes from a local process pool managed by the backend it-self: if we expect no new tasks, there is no point in re-creating a new working pool. Nr )r � ensure_readyr r r �abort_everythingE s z$ParallelBackendBase.abort_everything) �__name__� __module__� __qualname__�__doc__r r r r r r r r r r r r r r s r c @ s1 e Z d Z d Z d d � Z d d d � Z d S)�SequentialBackendz�A ParallelBackend which will execute all batches sequentially. Does not use/create any threading objects, and hence has minimal overhead. Used when n_jobs == 1. c C s | d k r t d � � d S)z?Determine the number of jobs which are going to run in parallelr z&n_jobs == 0 in Parallel has no meaningr )� ValueError)r r r r r r e s z"SequentialBackend.effective_n_jobsNc C s t | � } | r | | � | S)zSchedule a func to be run)�ImmediateResult)r r r �resultr r r r k s z SequentialBackend.apply_async)r r! r" r# r r r r r r r$ ^ s r$ c @ sL e Z d Z d Z d d � Z d d � Z d d d � Z d d d � Z d S) �PoolManagerMixinz,A helper class for managing pool of workers.c C sd | d k r t d � � nE t d k s3 | d k r7 d S| d k r` t t j � d | d � } | S)z?Determine the number of jobs which are going to run in parallelr z&n_jobs == 0 in Parallel has no meaningNr )r% r �max� cpu_count)r r r r r r v s z!PoolManagerMixin.effective_n_jobsc C s6 | j d k r2 | j j � | j j � d | _ d S)z#Shutdown the process or thread poolN)�_pool�closer )r r r r r � s zPoolManagerMixin.terminateNc C s | j j t | � d | �S)zSchedule a func to be runr )r+ r � SafeFunction)r r r r r r r � s z PoolManagerMixin.apply_asyncTc C s< | j � | r8 | j d | j j d | j | j j � d S)z@Shutdown the pool and restart a new one with the same parametersr r N)r r r r Z _backend_args)r r r r r r � s z!PoolManagerMixin.abort_everything)r r! r" r# r r r r r r r r r( s s r( c @ sF e Z d Z d Z d Z d Z d Z d Z d d � Z d d � Z d S) �AutoBatchingMixinz/A helper class for automagically batching jobs.g�������?� r g c C s | j } | j } | d k r� | | j k r� t | | j | � } t d | d � } | | _ | j j d k r� | j j d | | f � nb | | j k r� | d k r� | d } | | _ | j j d k r� | j j d | | f � n | } | | k rd | _ | S)z Determine the optimal batch sizer r/ r � z:Batch computation too fast (%.4fs.) Setting batch_size=%d.z:Batch computation too slow (%.4fs.) Setting batch_size=%d.) �_effective_batch_size�_smoothed_batch_duration�MIN_IDEAL_BATCH_DURATION�intr) r �verboseZ_print�MAX_IDEAL_BATCH_DURATION)r Zold_batch_sizeZbatch_durationZideal_batch_sizer r r r r � s2 z$AutoBatchingMixin.compute_batch_sizec C sL | | j k rH | j } | d k r- | } n d | d | } | | _ d S)z1Callback indicate how long it took to run a batchr g�������?g�������?N)r1 r2 )r r r Z old_durationZ new_durationr r r r � s z!AutoBatchingMixin.batch_completedN) r r! r" r# r3 r6 r1 r2 r r r r r r r. � s .r. c @ s( e Z d Z d Z d d d d � Z d S)�ThreadingBackenda� A ParallelBackend which will use a thread pool to execute batches in. This is a low-overhead backend but it suffers from the Python Global Interpreter Lock if the called function relies a lot on Python objects. Mostly useful when the execution bottleneck is a compiled extension that explicitly releases the GIL (for instance a Cython loop wrapped in a "with nogil" block or an expensive call to a library such as NumPy). r Nc K sF | j | � } | d k r* t t � � � | | _ t | � | _ | S)z?Build a process or thread pool and return the number of workersr )r �FallbackToBackendr$ r r r+ )r r r r r r r r � s zThreadingBackend.configure)r r! r" r# r r r r r r7 � s r7 c sR e Z d Z d Z d Z � f d d � Z d d d d � Z � f d d � Z � S) �MultiprocessingBackenda A ParallelBackend which will use a multiprocessing.Pool. Will introduce some communication and memory overhead when exchanging input and output data with the with the worker Python processes. However, does not suffer from the Python Global Interpreter Lock. Z__JOBLIB_SPAWNED_PARALLEL__c s� t d k r d St j � j rB | d k r> t j d d d �d St j � j d k rn t j d d d �d St t | � j | � S)z�Determine the number of jobs which are going to run in parallel. This also checks if we are attempting to create a nested parallel loop. Nr zHMultiprocessing-backed parallel loops cannot be nested, setting n_jobs=1� stacklevel� � MainThreadzVMultiprocessing backed parallel loops cannot be nested below threads, setting n_jobs=1) r �current_process�daemon�warnings�warn� threading�current_thread�name�superr9 r )r r )� __class__r r r s z'MultiprocessingBackend.effective_n_jobsr Nc K s� | j | � } | d k r* t t � � � t t j j | j d � � } | rZ t d � � d t j | j <t j � t | | � | _ | | _ | S)z?Build a process or thread pool and return the number of workersr r a% [joblib] Attempting to do parallel computing without protecting your import on a system that does not support forking. To use parallel-computing in a script, you must protect your main loop using "if __name__ == '__main__'". Please see the joblib documentation on Parallel for more information�1)r r8 r$ r4 �os�environ�get�JOBLIB_SPAWNED_PROCESS� ImportError�gcZcollectr r+ r )r r r r Zalready_forkedr r r r " s z MultiprocessingBackend.configurec s6 t t | � j � | j t j k r2 t j | j =d S)z#Shutdown the process or thread poolN)rD r9 r rJ rG rH )r )rE r r r ; s z MultiprocessingBackend.terminate)r r! r" r# rJ r r r r r )rE r r9 � s r9 c @ s( e Z d Z d d � Z d d � Z d S)r& c C s | � | _ d S)N)�results)r �batchr r r �__init__C s zImmediateResult.__init__c C s | j S)N)rM )r r r r rI H s zImmediateResult.getN)r r! r" rO rI r r r r r& B s r& c @ s. e Z d Z d Z d d � Z d d � Z d S)r- a Wrapper that handles the serialization of exception tracebacks. If an exception is triggered when calling the inner function, a copy of the full traceback is captured to make it possible to serialize it so that it can be rendered in a different Python process. c C s | | _ d S)N)r )r r r r r rO S s zSafeFunction.__init__c O s y | j | | � SWnd t k r1 t � � YnJ t j � \ } } } t | | | d d d d �} t | | � � Yn Xd S)N�contextr0 Z tb_offsetr )r �KeyboardInterruptr �sys�exc_infor r )r �args�kwargsZe_typeZe_valueZe_tb�textr r r �__call__V s zSafeFunction.__call__N)r r! r" r# rO rW r r r r r- L s r- c @ s" e Z d Z d Z d d � Z d S)r8 z<Raised when configuration should fallback to another backendc C s | | _ d S)N)�backend)r rX r r r rO g s zFallbackToBackend.__init__N)r r! r" r# rO r r r r r8 d s r8 )!r# rL rG rR r? rA �abcr r � format_stackr Z my_exceptionsr r Z_multiprocessing_helpersr Z_compatr �poolr Zmultiprocessing.poolr r r$ �objectr( r. r7 r9 r& r- � Exceptionr8 r r r r �<module> s. I"P G
Edit
Rename
Chmod
Delete
FILE
FOLDER
Name
Size
Permission
Action
__init__.cpython-35.opt-1.pyc
4779 bytes
0644
__init__.cpython-35.pyc
4779 bytes
0644
_compat.cpython-35.opt-1.pyc
677 bytes
0644
_compat.cpython-35.pyc
677 bytes
0644
_memory_helpers.cpython-35.opt-1.pyc
3223 bytes
0644
_memory_helpers.cpython-35.pyc
3223 bytes
0644
_multiprocessing_helpers.cpython-35.opt-1.pyc
922 bytes
0644
_multiprocessing_helpers.cpython-35.pyc
922 bytes
0644
_parallel_backends.cpython-35.opt-1.pyc
12908 bytes
0644
_parallel_backends.cpython-35.pyc
12908 bytes
0644
disk.cpython-35.opt-1.pyc
2571 bytes
0644
disk.cpython-35.pyc
2571 bytes
0644
format_stack.cpython-35.opt-1.pyc
9075 bytes
0644
format_stack.cpython-35.pyc
9123 bytes
0644
func_inspect.cpython-35.opt-1.pyc
9763 bytes
0644
func_inspect.cpython-35.pyc
9763 bytes
0644
hashing.cpython-35.opt-1.pyc
6653 bytes
0644
hashing.cpython-35.pyc
6653 bytes
0644
logger.cpython-35.opt-1.pyc
4184 bytes
0644
logger.cpython-35.pyc
4184 bytes
0644
memory.cpython-35.opt-1.pyc
28122 bytes
0644
memory.cpython-35.pyc
28122 bytes
0644
my_exceptions.cpython-35.opt-1.pyc
2750 bytes
0644
my_exceptions.cpython-35.pyc
2750 bytes
0644
numpy_pickle.cpython-35.opt-1.pyc
15536 bytes
0644
numpy_pickle.cpython-35.pyc
15536 bytes
0644
numpy_pickle_compat.cpython-35.opt-1.pyc
7262 bytes
0644
numpy_pickle_compat.cpython-35.pyc
7393 bytes
0644
numpy_pickle_utils.cpython-35.opt-1.pyc
16434 bytes
0644
numpy_pickle_utils.cpython-35.pyc
16434 bytes
0644
parallel.cpython-35.opt-1.pyc
25898 bytes
0644
parallel.cpython-35.pyc
25898 bytes
0644
pool.cpython-35.opt-1.pyc
18841 bytes
0644
pool.cpython-35.pyc
18841 bytes
0644
testing.cpython-35.opt-1.pyc
2657 bytes
0644
testing.cpython-35.pyc
2657 bytes
0644
N4ST4R_ID | Naxtarrr
