Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
How can Tensorflow be used to work with tf.data API and tokenizer?
The ‘tf.data’ API can be used to tokenize the strings. Tokenization is the method of breaking down a string into tokens. These tokens can be words, numbers, or punctuation.
Read More: What is TensorFlow and how Keras work with TensorFlow to create Neural Networks?
We will use the Keras Sequential API, which is helpful in building a sequential model that is used to work with a plain stack of layers, where every layer has exactly one input tensor and one output tensor.
A neural network that contains at least one layer is known as a convolutional layer. We can use the Convolutional Neural Network to build learning model.
TensorFlow Text contains collection of text related classes and ops that can be used with TensorFlow 2.0. The TensorFlow Text can be used to preprocess sequence modelling.
We are using the Google Colaboratory to run the below code. Google Colab or Colaboratory helps run Python code over the browser and requires zero configuration and free access to GPUs (Graphical Processing Units). Colaboratory has been built on top of Jupyter Notebook.
The important interfaces include Tokenizer and TokenizerWithOffsets each of which have a single method tokenize and tokenize_with_offsets respectively. There are multiple tokenizers, each of which implement TokenizerWithOffsets (which extends the Tokenizer class). This includes an option to get byte offsets into the original string. This helps know the bytes in the original string the token was created from.
Example
print("Tokenizer with tf.data API")
docs = tf.data.Dataset.from_tensor_slices([['Never tell me about the odds.'], ["It's not trye!"]])
print("Whitespace tokenizer is being called")
tokenizer = text.WhitespaceTokenizer()
tokenized_docs = docs.map(lambda x: tokenizer.tokenize(x))
iterator = iter(tokenized_docs)
print(next(iterator).to_list())
print(next(iterator).to_list())Code credit −https://www.tensorflow.org/tutorials/tensorflow_text/intro
Output
Tokenizer with tf.data API Whitespace tokenizer is being called [[b'Never', b'tell', b'me', b'about', b'the', b'odds.']] [[b"It's", b'not', b'trye!']]
Explanation
- The tf.data API is used with tokenizer.
- Once this is done, the whitespace tokenizer is called on the input.
- The tokens are iterated over, and then stored in a list.
137 Views
