profile
viewpoint

ghShu/CloudComparer 1

Compare the various managed cloud services offered by the major public cloud providers in the market.

ghShu/Algorithms 0

A collection of algorithms and data structures

ghShu/allaboutscala 0

Source code for www.allaboutscala.com tutorials

ghShu/awesome-courses 0

:books: List of awesome university courses for learning Computer Science!

ghShu/charts 0

Helm Charts

ghShu/cheatsheet-translation 0

Translation of VIP cheatsheets for Machine Learning Deep Learning, and Artificial Intelligence

ghShu/cheatsheets 0

My cheatsheets

ghShu/cloud-demos 0

Building cloud native services with simple examples

issue commenttensorflow/tensorflow

GPU and CPU utilization slashed after first epoch using tf.data and tf.keras

I tried generating synthetic data to replicate the problem but was unable to (1.2M simple JPEGs at 6GB total is not equivalent to 1.2M JPEGs occupying 160GB as with the ImageNet dataset).

However, I got Tensorboard profiling up and captured some very interesting traces that highlight the problem.

Epoch 1 overview (most of the time is spent computing on the GPU):

Epoch1_overview

Epoch 2 overview (input bottleneck appears):

Epoch2_overview

What's going on?

Trace view indicates that in Epoch 1, data loading for the subsequent batch happens across four threads during training of the current batch. Loading a batch takes <50% of the training time, so by the time the current batch is completed, it is ready to proceed to the next.

Here is Epoch 1's trace:

Epoch1_prefetch

Zooming in, we can see that the loading operation consists of reading, JPEG decoding, a cast, and a resize. They happen one after another.

Epoch1_prefetch_mid_training

Now compare Epoch 2:

Epoch2_saturated

We can see the input pipeline has slowed down dramatically and it takes longer than a training step to load a batch.

<em>Why are their gaps between the different input pipeline steps in Epoch 2?</em> There is idle time between reading the file, decoding the JPEG, resizing, etc.

<em>tf_data_iterator_get_next is consuming more time.</em> Why? It is barely visible in Epoch 1's trace.

Is this something to do with the way my pipeline is constructed?

Here is Epoch 2's input bottleneck analysis but I am unsure how to interpret it because I have only a map() call, not map_and_batch as shown here:

Epoch2_bottleneck_analysis

I hope this is more helpful.

trzy

comment created time in 4 minutes

issue commenttensorflow/tensorflow

How to build static libtensorflow_cc.a and libtensorflow_framework.a for Linux and MacOS

I was able to make some things work by using the object files reported in the *.params:

ar -cr libtensorflow.a $(cat bazel-bin/tensorflow/libtensorflow_*.so.*.params | grep '\.o$')

... and then link with -Wl,--allow-multiple-definition -Wl,--whole-archive libtensorflow.a

Note: I first tried creating separate static libraries corresponding to the shared libraries, libtensorflow_cc.a and libtensorflow_framework.a as noted in this issue's title, but encountered a lot of problems with that approach (e.g. duplicate registrations). Putting it all in one archive seemed to magically resolve that.

Hope this helps someone!

samhodge

comment created time in 8 minutes

issue commenttensorflow/tensorflow

Keras to tflite. Multiple outputs all given name Identity

Same issue here, All outputs ordered after their defined names but shows as Identity

marno1d

comment created time in 8 minutes

push eventtensorflow/tensorflow

Tres Popp

commit sha 5f6c24a343b45934415c5f6f76a16d5e1cd6612b

Create a test for every tf_to_kernel invocation. This additionally will create a nice reproducer command on any failure. PiperOrigin-RevId: 344310249 Change-Id: Id076d95a80430f4f71f3fa091c4e7943e981b83f

view details

push time in 11 minutes

push eventtensorflow/tensorflow

Anna R

commit sha c498ad75ddabf49c1b470ee8d9ae583a23be44b1

Only read *.pbtxt files in api_def/*_api directories when generating Go API. PiperOrigin-RevId: 344308260 Change-Id: I1563382acd8eda94f2b46f92d2957522e1d5b79e

view details

push time in 21 minutes

Pull request review commenttensorflow/tensorflow

Freeze improvements

 def convert_variables_to_constants(sess,                                    input_graph_def,                                    output_node_names,                                    variable_names_whitelist=None,-                                   variable_names_blacklist=None):+                                   variable_names_blacklist=None,+                                   check_and_revert_if_type_mismatch=False):

Can we document what this new argument does? Ideally with a usage example. perhaps the name could be more self explanatory, right now it's not clear from the name what it does.

Also, when would someone want this to be False? If no one should want this to be False we can just make the change and not have it as an option.

sboshin

comment created time in 39 minutes

issue commenttensorflow/tensorflow

ValueError: All inputs to `ConcreteFunction`s must be Tensors

Are you satisfied with the resolution of your issue? <a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/44649">Yes</a> <a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/44649">No</a>

moha23

comment created time in 36 minutes

issue closedtensorflow/tensorflow

ValueError: All inputs to `ConcreteFunction`s must be Tensors

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): linux 20.04
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary): pip
  • TensorFlow version (use command below): 2.1.2
  • Python version: 3.6
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: 10.1/7.4
  • GPU model and memory: gtx1080Ti

Describe the current behavior Tf.train_on_batch cant take numpy array as input, asks for tensor and even using tf.convert_to_tensor doesn't help. It works in Tf2.3, but Tf2.1 is a requirement for compatibility with another library.

Describe the expected behavior Should train without issues, tested on Tf2.3 and it works, but Tf2.1 is a requirement.

Standalone code to reproduce the issue

import tensorflow as tf
import numpy as np
from sklearn.model_selection import train_test_split
from tensorflow.keras.layers import (Conv2D,Input)
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
import tensorflow_addons as tfa #version 0.9.1

opt = Adam(lr=1e-4)
tf.config.experimental_run_functions_eagerly(True)


def custom_mean_squared_error(y_true, y_pred):
    return(tf.reduce_mean(tf.math.squared_difference(y_true, y_pred)))


def firstStream(ip):
        layer = Conv2D(64, kernel_size=(5, 5), padding='same')(ip)
        op = Conv2D(3, (5, 5), activation='sigmoid', padding='same')(layer)
        return op
    
def secStream(ip):
        layer = Conv2D(8, kernel_size=(5, 5), padding='same')(ip)
        flow = Conv2D(2, (5, 5), activation='sigmoid', padding='same')(layer)
        return flow


def main():

    ip = Input(shape=(None, None, 3))
    op = firstStream(ip)
    flow = secStream(ip)
    warped = tf.expand_dims(tfa.image.dense_image_warp(op,flow,name='warp'),axis=1)
    model = Model(ip,warped)
    model.compile(optimizer=opt, loss=custom_mean_squared_error)
        
    numEpochs = 1

    for epochNo in range(numEpochs):

            print("Running epoch : %d" % epochNo)
            batch_ip=np.ones((1,64,64,3))
                  
            curr_loss = model.train_on_batch(batch_ip, batch_ip)
            print(curr_loss)


main()

Other info / logs

main() File "/home/mohana/work/simple.py", line 46, in main curr_loss = model.train_on_batch(batch_ip, batch_ip) File "/home/mohana/virtualenvs/tfcomp/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 1078, in train_on_batch standalone=True) File "/home/mohana/virtualenvs/tfcomp/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 433, in train_on_batch output_loss_metrics=model._output_loss_metrics) File "/home/mohana/virtualenvs/tfcomp/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 312, in train_on_batch output_loss_metrics=output_loss_metrics)) File "/home/mohana/virtualenvs/tfcomp/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 269, in _process_single_batch grads = tape.gradient(scaled_total_loss, trainable_weights) File "/home/mohana/virtualenvs/tfcomp/lib/python3.6/site-packages/tensorflow_core/python/eager/backprop.py", line 1029, in gradient unconnected_gradients=unconnected_gradients) File "/home/mohana/virtualenvs/tfcomp/lib/python3.6/site-packages/tensorflow_core/python/eager/imperative_grad.py", line 77, in imperative_grad compat.as_str(unconnected_gradients.value)) File "/home/mohana/virtualenvs/tfcomp/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1256, in _backward_function_wrapper processed_args, remapped_captures) File "/home/mohana/virtualenvs/tfcomp/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1684, in _call_flat "Tensor." % (self._func_graph.name, i, str(arg))) ValueError: All inputs to ConcreteFunctions must be Tensors; on invocation of __backward__defun_call_1331, the 0-th input (IndexedSlices(indices=tf.Tensor([ 65 65 66 ... 4028 4029 4030], shape=(16384,), dtype=int32), values=tf.Tensor( [[-0.0000000e+00 -0.0000000e+00 -0.0000000e+00] [-0.0000000e+00 -0.0000000e+00 -0.0000000e+00] [-0.0000000e+00 -0.0000000e+00 -0.0000000e+00] ... [-2.7218828e-05 -2.6730117e-05 -2.4183297e-05] [-2.7333013e-05 -2.7423557e-05 -2.2896325e-05] [-2.1508897e-05 -2.1834361e-05 -1.8788463e-05]], shape=(16384, 3), dtype=float32), dense_shape=tf.Tensor([4096 3], shape=(2,), dtype=int32))) was not a Tensor.

closed time in 36 minutes

moha23

issue commenttensorflow/tensorflow

ValueError: All inputs to `ConcreteFunction`s must be Tensors

Closing as stale. Please reopen if you'd like to work on this further.

moha23

comment created time in 36 minutes

pull request commenttensorflow/tensorflow

Always build libtensorflow in a manylinux2010 compatible way

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

<!-- need_sender_cla -->

VivekPanyam

comment created time in 44 minutes

PR opened tensorflow/tensorflow

Always build libtensorflow in a manylinux2010 compatible way

Previously, only the CUDA builds of libtensorflow for TF 2.x would be built in a manylinux2010 sysroot.

This PR builds the CPU version in a manylinux2010 sysroot as well.

+3 -1

0 comment

2 changed files

pr created time in 44 minutes

Pull request review commenttensorflow/tensorflow

Serializing EagerTensors in model.save()

 from tensorflow.python.framework import dtypes from tensorflow.python.framework import tensor_shape from tensorflow.python.util.compat import collections_abc-+import tensorflow as tf

Please use from tensorflow.python like the other imports

stratomaster31

comment created time in an hour

Pull request review commenttensorflow/tensorflow

Serializing EagerTensors in model.save()

 def get_json_type(obj):   # e.g. optimizer, layer   if hasattr(obj, 'get_config'):     return {'class_name': obj.__class__.__name__, 'config': obj.get_config()}+  +  # Serialize EagerTensors (fix for saving 1.x Keras Models)+  if isinstance(obj, tf.python.framework.ops.EagerTensor):

You need to create a new test for this feature at https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/tests/serialization_util_test.py

stratomaster31

comment created time in an hour

issue commenttensorflow/tensorflow

Failed build pip package on MacOS Big Sur

I sent you only part of my screen that contains the error and nothing else. Okay, but when I didn't found a wheel, I intuitively a try to build my own wheel by instructions posted here: https://www.tensorflow.org/install/source .... Thanks.

markub3327

comment created time in an hour

push eventtensorflow/tensorflow

Edward Loper

commit sha a7eb0f4531922936f2bfec95978e14ae3f60294a

Type converter for Python API parameters. PiperOrigin-RevId: 344302045 Change-Id: I5a1a2f1ef59660882506a0a28d979aa61b5fc028

view details

push time in an hour

push eventtensorflow/tensorflow

Ce Zheng

commit sha 185e785f9ad8d261c07ae0defa45acc945fad2ee

[XLA] Add a pass to fold hlo(convert(a),convert(b)) into hlo(a,b) to enable an alternative way of specifying wider accumulation type than the shape inference result for HLOs that support it. PiperOrigin-RevId: 344301743 Change-Id: If3b84a8369fd396012a6915237610e10f5e0d318

view details

push time in an hour

push eventtensorflow/tensorflow

Rick Chao

commit sha 287d8116bf05d8f1468580d7349740c0dbb3a1b5

PSv2: Add checks that `ParameterServerStrategy`'s `run`, `reduce`, `experimental_distribute_dataset`, and `distribute_datasets_from_function` are used with a `ClusterCoordinator`, and that `run` and `reduce` need to be used within a function that is used with `schedule`. PiperOrigin-RevId: 344297761 Change-Id: Ia7c101ed04e2db24564e83b47db7762b096ec267

view details

push time in 2 hours

push eventtensorflow/tensorflow

Raman Sarokin

commit sha cc9e5d8075fa57cd17126e0a5faf9846340caf20

Relu test logic moved to gpu/common/tasks. PiperOrigin-RevId: 344296799 Change-Id: Ic033efa5401afd8823f48f7ad4ce47c720aaa66e

view details

push time in 2 hours

PR closed tensorflow/tensorflow

Reviewers
Add idempotent based folding to TF cla: yes size:M stalled stat:awaiting response

This trait label will allow idempotent optimizations (i.e. multiples copies of op folded to one copy) to run automatically from the MLIR side

+201 -11

4 comments

2 changed files

ahmedsabie

pr closed time in 2 hours

issue commenttensorflow/tensorflow

TensorFlow Lite error on iOS: "Make sure you apply/link the Flex delegate before inference."

@yyoon thanks for the suggestion.

I am not very familiar with Xcode and I'm not sure how to tell what mode I'm running the app in. I tried the various options under the "play" button, but I get the same error:

Screen Shot 2020-11-25 at 12 06 00 PM

Also, I should add that I made sure to include the linker flags for both "debug" and "release" under build settings.

alexdmiller

comment created time in 2 hours

push eventtensorflow/tensorflow

A. Unique TensorFlower

commit sha 528d6545c21e6abd37788b7324355e1b094f7daf

Add CropAndResize cost estimate PiperOrigin-RevId: 344292836 Change-Id: I711a96bb300bbff52ea2219b9055ac0f3993b33f

view details

push time in 2 hours

push eventtensorflow/tensorflow

Francois Chollet

commit sha e2592b25ceaede4409aeb04343b8924b4449e03a

Fix minor docstring formatting issue PiperOrigin-RevId: 344290636 Change-Id: I63e22915e555ba4547f43fcd035298af9c597eb7

view details

push time in 2 hours

issue commenttensorflow/tensorflow

Missing wheels tagged for macosx_11_x

Ultimately yes, but we want to change how bdist is generated so we don't have to rename manually.

dgladkov

comment created time in 2 hours

issue closedtensorflow/tensorflow

443

Thank you for submitting a TensorFlow documentation issue. Per our GitHub policy, we only address code/doc bugs, performance issues, feature requests, and build/installation issues on GitHub.

The TensorFlow docs are open source! To get involved, read the documentation contributor guide: https://www.tensorflow.org/community/contribute/docs

URL(s) with the issue:

Please provide a link to the documentation entry, for example: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/MyMethod

Description of issue (what needs changing):

Clear description

For example, why should someone use this method? How is it useful?

Correct links

Is the link to the source code correct?

Parameters defined

Are all parameters defined and formatted correctly?

Returns defined

Are return values defined?

Raises listed and defined

Are the errors defined? For example, https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/feature_column/categorical_column_with_vocabulary_file#raises

Usage example

Is there a usage example?

See the API guide: https://www.tensorflow.org/community/contribute/docs_ref on how to write testable usage examples.

Request visuals, if applicable

Are there currently visuals? If not, will it clarify the content?

Submit a pull request?

Are you planning to also submit a pull request to fix the issue? See the docs contributor guide: https://www.tensorflow.org/community/contribute/docs, docs API guide: https://www.tensorflow.org/community/contribute/docs_ref and the docs style guide: https://www.tensorflow.org/community/contribute/docs_style

closed time in 2 hours

lintrin

issue openedtensorflow/tensorflow

443

Thank you for submitting a TensorFlow documentation issue. Per our GitHub policy, we only address code/doc bugs, performance issues, feature requests, and build/installation issues on GitHub.

The TensorFlow docs are open source! To get involved, read the documentation contributor guide: https://www.tensorflow.org/community/contribute/docs

URL(s) with the issue:

Please provide a link to the documentation entry, for example: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/MyMethod

Description of issue (what needs changing):

Clear description

For example, why should someone use this method? How is it useful?

Correct links

Is the link to the source code correct?

Parameters defined

Are all parameters defined and formatted correctly?

Returns defined

Are return values defined?

Raises listed and defined

Are the errors defined? For example, https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/feature_column/categorical_column_with_vocabulary_file#raises

Usage example

Is there a usage example?

See the API guide: https://www.tensorflow.org/community/contribute/docs_ref on how to write testable usage examples.

Request visuals, if applicable

Are there currently visuals? If not, will it clarify the content?

Submit a pull request?

Are you planning to also submit a pull request to fix the issue? See the docs contributor guide: https://www.tensorflow.org/community/contribute/docs, docs API guide: https://www.tensorflow.org/community/contribute/docs_ref and the docs style guide: https://www.tensorflow.org/community/contribute/docs_style

created time in 2 hours

push eventtensorflow/tensorflow

Yunxing Dai

commit sha 64ccd768cbe32c802144f66a795ac9b5e6c821f5

Don't rewrite operand's ancestors into predicates in dynamism inference. We don't rewrite predicates of kSelect and indices of kGather into predicates, and we also don't want to rewrite their ancestors into predicates. PiperOrigin-RevId: 344287633 Change-Id: I0dcb42023a84ebd6d1e0fd29f14a152255aed1bf

view details

push time in 2 hours

issue commenttensorflow/tensorflow

3080 & 3090 coumpute capability 86 degraded performance after some updates

Retested on 20.11 container. 3080 performance still effed up https://fsymbols.com/3080-3090-benchmarks/

ibmua

comment created time in 3 hours

issue commenttensorflow/tensorflow

TensorFlow 2.4 & roadmap

Thank you for your support.

peter197321

comment created time in 3 hours

PR merged tensorflow/tensorflow

Reviewers
ifdef out the known failing test cases for FullyConnected for hifimini. cla: yes comp:micro ready to pull size:M

Manually confirmed that the following command passes:

make -f tensorflow/lite/micro/tools/make/Makefile -j8 TARGET=xtensa OPTIMIZED_KERNEL_DIR=xtensa TARGET_ARCH=hifimini XTENSA_CORE=mini1m1m_RG test_kernel_fully_connected_test

http://b/170503075

+53 -54

2 comments

1 changed file

advaitjain

pr closed time in 3 hours

more