Submit
Path:
~
/
/
proc
/
thread-self
/
root
/
opt
/
alt
/
python35
/
lib64
/
python3.5
/
site-packages
/
sklearn
/
datasets
/
__pycache__
/
File Content:
twenty_newsgroups.cpython-35.opt-1.pyc
��(X�5 � @ s� d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l j Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z m Z e j rd d l m Z n d d l m Z e j e � Z! d Z" d Z# d Z$ d Z% d Z& d d � Z' d d � Z( e j) d � Z* d d � Z+ d d � Z, d d d d d f d d d � Z- d f d d! d" � Z. d S)#a| Caching loader for the 20 newsgroups text classification dataset The description of the dataset is available on the official website at: http://people.csail.mit.edu/jrennie/20Newsgroups/ Quoting the introduction: The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder: Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering. This dataset loader will download the recommended "by date" variant of the dataset and which features a point in time split between the train and test sets. The compressed dataset size is around 14 Mb compressed. Once uncompressed the train set is 52 MB and the test set is 34 MB. The data is downloaded, extracted and cached in the '~/scikit_learn_data' folder. The `fetch_20newsgroups` function will not vectorize the data into numpy arrays but the dataset lists the filenames of the posts and their categories as target labels. The `fetch_20newsgroups_vectorized` function will in addition do a simple tf-idf vectorization step. � N� )� get_data_home)�Bunch)� load_files)� _pkl_filepath� )�check_random_state)�CountVectorizer)� normalize)�joblib�six)�urlopenzEhttp://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gzz20news-bydate.tar.gzz20news-bydate.pkzz20news-bydate-trainz20news-bydate-testc C s� t j j | t � } t j j | t � } t j j | t � } t j j | � s^ t j | � t j j | � r� t j d � t j | � t j d t � t t � } t | d � � } | j | j � � Wd QRXt j d | � t j | d � j d | � t j | � t d t | d d �d t | d d �� } t j t j | � d � } t | d � � } | j | � Wd QRXt j | � | S) zADownload the 20 newsgroups data and stored it as a zipped pickle.z+Download was incomplete, downloading again.z#Downloading dataset from %s (14 MB)�wbNzDecompressing %szr:gz�path�train�encoding�latin1�test� zlib_codec)�osr �join� ARCHIVE_NAME� TRAIN_FOLDER� TEST_FOLDER�exists�makedirs�logger�warning�remove�URLr �open�write�read�info�tarfile� extractall�dictr �codecs�encode�pickle�dumps�shutil�rmtree) � target_dir� cache_pathZ archive_pathZ train_pathZ test_pathZopener�f�cache�compressed_content� r2 �/twenty_newsgroups.py�download_20newsgroupsK s, r4 c C s | j d � \ } } } | S)zq Given text in "news" format, strip the headers, by removing everything before the first blank line. z )� partition)�textZ_beforeZ _blankline�afterr2 r2 r3 �strip_newsgroup_headern s r8 zF(writes in|writes:|wrote:|says:|said:|^In article|^Quoted from|^\||^>)c C s) d d � | j d � D� } d j | � S)z� Given text in "news" format, strip lines beginning with the quote characters > or |, plus lines that often introduce a quoted section (for example, because they contain the string 'writes:'.) c S s% g | ] } t j | � s | � q Sr2 )� _QUOTE_RE�search)�.0�liner2 r2 r3 � <listcomp>� s z+strip_newsgroup_quoting.<locals>.<listcomp>� )�splitr )r6 Z good_linesr2 r2 r3 �strip_newsgroup_quoting{ s r@ c C s� | j � j d � } xJ t t | � d d d � D], } | | } | j � j d � d k r2 Pq2 W| d k r� d j | d | � � S| Sd S) a Given text in "news" format, attempt to remove a signature block. As a rough heuristic, we assume that signatures are set apart by either a blank line or a line made of hyphens, and that it is the last such line in the file (disregarding blank lines at the end). r> r �-� r N���rC )�stripr? �range�lenr )r6 �linesZline_numr< r2 r2 r3 �strip_newsgroup_footer� s # rH r T�* c s� t d | � } t | t � } t j j | d � } d } t j j | � r� yJ t | d � � } | j � } Wd QRXt j | d � } t j | � } WnU t k r� } z5 t d d � t d � t d d � t | � WYd d } ~ Xn X| d k r0| r$t j d � t d | d | � } n t d � � | d k rI| | � n� | d k r�t � } t � } t � } xH d D]@ } | | � | j � j � | j � j � | j � j � qwW| � _ t j | � � _ t j | � � _ n t d | � � d � _ d | k r)d d � � j D� � _ d | k rNd d � � j D� � _ d | k rsd d � � j D� � _ | d k rA� f d d � | D� } | j � t | � \ } } t j � j | � } � j | � _ � j | � _ t j | � j � � _ t | � � _ t j � j d t! �} | | } | j" � � _ | r�t# | � } t j$ � j j% d � } | j& | � � j | � _ � j | � _ t j � j d t! �} | | } | j" � � _ � S) a& Load the filenames and data from the 20 newsgroups dataset. Read more in the :ref:`User Guide <20newsgroups>`. Parameters ---------- subset : 'train' or 'test', 'all', optional Select the dataset to load: 'train' for the training set, 'test' for the test set, 'all' for both, with shuffled ordering. data_home : optional, default: None Specify a download and cache folder for the datasets. If None, all scikit-learn data is stored in '~/scikit_learn_data' subfolders. categories : None or collection of string or unicode If None (default), load all the categories. If not None, list of category names to load (other categories ignored). shuffle : bool, optional Whether or not to shuffle the data: might be important for models that make the assumption that the samples are independent and identically distributed (i.i.d.), such as stochastic gradient descent. random_state : numpy random number generator or seed integer Used to shuffle the dataset. download_if_missing : optional, True by default If False, raise an IOError if the data is not locally available instead of trying to download the data from the source site. remove : tuple May contain any subset of ('headers', 'footers', 'quotes'). Each of these are kinds of text that will be detected and removed from the newsgroup posts, preventing classifiers from overfitting on metadata. 'headers' removes newsgroup headers, 'footers' removes blocks at the ends of posts that look like signatures, and 'quotes' removes lines that appear to be quoting another post. 'headers' follows an exact standard; the other filters are not always correct. � data_homeZ 20news_homeN�rbr �P �_zCache loading failedz8Downloading 20news dataset. This may take a few minutes.r- r. z 20Newsgroups dataset not foundr r �allz5subset can only be 'train', 'test' or 'all', got '%s'z!the 20 newsgroups by date datasetZheadersc S s g | ] } t | � � q Sr2 )r8 )r; r6 r2 r2 r3 r= � s z&fetch_20newsgroups.<locals>.<listcomp>Zfootersc S s g | ] } t | � � q Sr2 )rH )r; r6 r2 r2 r3 r= � s Zquotesc S s g | ] } t | � � q Sr2 )r@ )r; r6 r2 r2 r3 r= � s c s( g | ] } � j j | � | f � q Sr2 )� target_names�index)r; �cat)�datar2 r3 r= s �dtyper )r r )r r )'r r � CACHE_NAMEr r r r r r" r' �decoder) �loads� Exception�printr r# r4 �IOError�list�extendrR �target� filenames�np�array� ValueError� description�sort�zipZin1dZ searchsortedrO �object�tolistr Zarange�shape�shuffle)rJ �subset� categoriesrg � random_stater Zdownload_if_missingr. Z twenty_homer0 r/ r1 Zuncompressed_content�eZdata_lstr\ r] Zlabels�mask�indicesr2 )rR r3 �fetch_20newsgroups� s� 1 rn c C s t d | � } d } | r2 | d d j | � 7} t | | d � } t d | d d d d d d d d d | � } t d | d d d d d d d d d | � } t j j | � r� t j | � \ } } n^ t d t j � } | j | j � j � } | j | j � j � } t j | | f | d d �| j t j � } | j t j � } t | d d �t | d d �| j } | d k r�| } | j } ns | d k r�| } | j } nU | d k r�t j | | f � j � } t j | j | j f � } n t d | � � t d | d | d | � S)a� Load the 20 newsgroups dataset and transform it into tf-idf vectors. This is a convenience function; the tf-idf transformation is done using the default settings for `sklearn.feature_extraction.text.Vectorizer`. For more advanced usage (stopword filtering, n-gram extraction, etc.), combine fetch_20newsgroups with a custom `Vectorizer` or `CountVectorizer`. Read more in the :ref:`User Guide <20newsgroups>`. Parameters ---------- subset : 'train' or 'test', 'all', optional Select the dataset to load: 'train' for the training set, 'test' for the test set, 'all' for both, with shuffled ordering. data_home : optional, default: None Specify an download and cache folder for the datasets. If None, all scikit-learn data is stored in '~/scikit_learn_data' subfolders. remove : tuple May contain any subset of ('headers', 'footers', 'quotes'). Each of these are kinds of text that will be detected and removed from the newsgroup posts, preventing classifiers from overfitting on metadata. 'headers' removes newsgroup headers, 'footers' removes blocks at the ends of posts that look like signatures, and 'quotes' removes lines that appear to be quoting another post. Returns ------- bunch : Bunch object bunch.data: sparse matrix, shape [n_samples, n_features] bunch.target: array, shape [n_samples] bunch.target_names: list, length [n_classes] rJ Z20newsgroup_vectorizedzremove-rA z.pklrh r ri Nrg Trj � r r rS �compress� �copyFrN zC%r is not a valid subset: should be one of ['train', 'test', 'all']rR r\ rO )r r r rn r r r r �loadr r^ Zint16Z fit_transformrR ZtocsrZ transform�dumpZastypeZfloat64r rO r\ �spZvstackZ concatenater` r ) rh r rJ ZfilebaseZ target_fileZ data_trainZ data_testZX_trainZX_testZ vectorizerrO rR r\ r2 r2 r3 � fetch_20newsgroups_vectorized sP ' rv )/�__doc__r �loggingr$ r) r+ �rer' Znumpyr^ Z scipy.sparseZsparseru �baser r r r Zutilsr Zfeature_extraction.textr Z preprocessingr Z externalsr r ZPY3Zurllib.requestr �urllib2� getLogger�__name__r r r rT r r r4 r8 �compiler9 r@ rH rn rv r2 r2 r2 r3 �<module>" sH # �
Submit
FILE
FOLDER
Name
Size
Permission
Action
__init__.cpython-35.opt-1.pyc
3078 bytes
0644
__init__.cpython-35.pyc
3078 bytes
0644
base.cpython-35.opt-1.pyc
24686 bytes
0644
base.cpython-35.pyc
24686 bytes
0644
california_housing.cpython-35.opt-1.pyc
3623 bytes
0644
california_housing.cpython-35.pyc
3623 bytes
0644
covtype.cpython-35.opt-1.pyc
3595 bytes
0644
covtype.cpython-35.pyc
3595 bytes
0644
kddcup99.cpython-35.opt-1.pyc
12034 bytes
0644
kddcup99.cpython-35.pyc
12034 bytes
0644
lfw.cpython-35.opt-1.pyc
16547 bytes
0644
lfw.cpython-35.pyc
16547 bytes
0644
mlcomp.cpython-35.opt-1.pyc
3126 bytes
0644
mlcomp.cpython-35.pyc
3126 bytes
0644
mldata.cpython-35.opt-1.pyc
6339 bytes
0644
mldata.cpython-35.pyc
6339 bytes
0644
olivetti_faces.cpython-35.opt-1.pyc
4514 bytes
0644
olivetti_faces.cpython-35.pyc
4514 bytes
0644
rcv1.cpython-35.opt-1.pyc
7184 bytes
0644
rcv1.cpython-35.pyc
7184 bytes
0644
samples_generator.cpython-35.opt-1.pyc
51364 bytes
0644
samples_generator.cpython-35.pyc
51364 bytes
0644
setup.cpython-35.opt-1.pyc
814 bytes
0644
setup.cpython-35.pyc
814 bytes
0644
species_distributions.cpython-35.opt-1.pyc
7434 bytes
0644
species_distributions.cpython-35.pyc
7434 bytes
0644
svmlight_format.cpython-35.opt-1.pyc
15631 bytes
0644
svmlight_format.cpython-35.pyc
15631 bytes
0644
twenty_newsgroups.cpython-35.opt-1.pyc
12307 bytes
0644
twenty_newsgroups.cpython-35.pyc
12307 bytes
0644
N4ST4R_ID | Naxtarrr
