How can Tensorflow and Python be used to build ragged tensor from list of words?

A RaggedTensor can be built by using the starting offsets of the words in the sentence. Firstly, the code point of every character in every word in the sentence is built. Next, they are displayed on the console. The number of words in that specific sentence is determined, and the offset is determined.

Read More: What is TensorFlow and how Keras work with TensorFlow to create Neural Networks?

Represent Unicode strings using Python, and manipulate those using Unicode equivalents. At first, we will separate the Unicode strings into tokens based on script detection with the help of the Unicode equivalents of standard string ops.

We are using the Google Colaboratory to run the below code. Google Colab or Colaboratory helps run Python code over the browser and requires zero configuration and free access to GPUs (Graphical Processing Units). Colaboratory has been built on top of Jupyter Notebook.

print("Get the code point of every character in every word")
word_char_codepoint = tf.RaggedTensor.from_row_starts(
   values=sentence_char_codepoint.values,
   row_starts=word_starts)
print(word_char_codepoint)
print("Get the number of words in the specific sentence")
sentence_num_words = tf.reduce_sum(tf.cast(sentence_char_starts_word, tf.int64), axis=1)

Code credit: https://www.tensorflow.org/tutorials/load_data/unicode

Output

Get the code point of every character in every word
<tf.RaggedTensor [[72, 101, 108, 108, 111], [44, 32], [116, 104, 101, 114, 101], [46], [19990, 30028], [12371, 12435, 12395, 12385, 12399]]>
Get the number of words in the specific sentence

Explanation

• The code point for every character in every word is built.
• These are displayed on the console.
• The number of words in that specific sentence is determined.

AmitDiwan

Updated on 20-Feb-2021 06:55:35

• Related Questions & Answers
• How can Tensorflow be used to build vocabulary from tokenized words for Illiad dataset using Python?
• How can Tensorflow used to segment word code point of ragged tensor back to sentences?
• How can Tensorflow be used to build normalization layer using Python?
• How can TensorFlow be used to create a tensor and display a message using Python?
• After normalization, how can Tensorflow be used to train and build the model?
• How can Tensorflow be used to build a one dimensional convolutional network using Python?
• How can TensorFlow be used to build the model for Fashion MNIST dataset in Python?
• How can Tensorflow be used with abalone dataset to build a sequential model?
• How can Tensorflow be used to convert the tokenized words from Illiad dataset into integers using Python?
• How can Tensorflow be used with pre-trained model to build the training and validation dataset?
• How can Tensorflow be used to build a normalization layer for the abalone dataset?
• How can Tensorflow and Tensorflow text be used to tokenize string data?
• How can Tensorflow and Python be used to verify the CIFAR dataset?
• How can augmentation be used to reduce overfitting using Tensorflow and Python?
• How can Tensorflow be used to compose layers using Python?