Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
How can Tensorflow be used to convert the tokenized words from Illiad dataset into integers using Python?
Tensorflow is a machine learning framework that is provided by Google. It is an open-source framework used in conjunction with Python to implement algorithms, deep learning applications and much more. It is used in research and for production purposes.
The ‘tensorflow’ package can be installed on Windows using the below line of code −
pip install tensorflow
Tensor is a data structure used in TensorFlow. It helps connect edges in a flow diagram. This flow diagram is known as the ‘Data flow graph’. Tensors are nothing but a multidimensional array or a list.
They can be identified using three main attributes −
Rank − It tells about the dimensionality of the tensor. It can be understood as the order of the tensor or the number of dimensions in the tensor that has been defined.
Type − It tells about the data type associated with the elements of the Tensor. It can be a one dimensional, two dimensional or n-dimensional tensor.
Shape − It is the number of rows and columns together.
We will be using the Illiad’s dataset, which contains text data of three translation works from William Cowper, Edward (Earl of Derby) and Samuel Butler. The model is trained to identify the translator when a single line of text is given. The text files used have been preprocessing. This includes removing the document header and footer, line numbers and chapter titles.
We are using Google Colaboratory to run the below code. Google Colab or Colaboratory helps run Python code over the browser and requires zero configuration and free access to GPUs (Graphical Processing Units). Colaboratory has been built on top of Jupyter Notebook.
Example
Following is the code snippet −
keys = vocab
values = range(2, len(vocab) + 2) # reserve 0 for padding, 1 for OOV
print("Map the tokens to integers")
init = tf.lookup.KeyValueTensorInitializer(
keys, values, key_dtype=tf.string, value_dtype=tf.int64)
num_oov_buckets = 1
vocab_table = tf.lookup.StaticVocabularyTable(init, num_oov_buckets)
print("A function has been defined to standardize, tokenize and vectorize the dataset using
tokenizer and lookup table")
def preprocess_text(text, label):
standardized = tf_text.case_fold_utf8(text)
tokenized = tokenizer.tokenize(standardized)
vectorized = vocab_table.lookup(tokenized)
return vectorized, labelCode credit − https://www.tensorflow.org/tutorials/load_data/text
Output
Map the tokens to integers A function has been defined to standardize, tokenize and vectorize the dataset using tokenizer and lookup table
Explanation
The vocab set is used to create a StaticVocabularyTable.
The tokens are mapped to integers within the range [2, vocab_size + 2].
The number 0 is used to indicate padding and 1 is used to indicate an out-of-vocabulary (OOV) token.
71 Views
